Netpeak Spider can be a desktop Software that can be used to crawl any website (similar to the process utilized by a online search engine). Learners and academic institutions may well qualify for Specific educational licenses made available from Screaming Frog. Examine their eligibility standards to view If you're able https://romeotools.com/product/screaming-frog/