Automated Web Scraper in 2 minutes.. inspired by F. W. Taylor
F.W. Taylor aka ‘The Father of Scientific Management’ inspired a scientific method to productivity improvement which revolutionized the industry. Why not implement scientific management in market research as well? Limit your manual Google search henceforth...
While carrying out any market research/research for a thesis, its difficult to keep a track record of the links and the content you manually scan during the entire research. This makes you less efficient, searching manually through any search engine is unproductive and boring. What’s the solution then?
I bring you Automated Web Scraper Tool. This tool can google search for links pertaining to your search query and scrape content from the websites crawled with the google search engine, while you make your favorite Dalgona coffee.
The scraping of website is done using Newspaper3k python library. I am attaching its documentation below-
The Web Crawling is done using Googlesearch python library-
Both of these have been combined into a loop to give a Table which is a scrape tool with no worries. Here is the snapshot of a sample result Table for the search query-Data Science.
This will help standardize your research and reduce your workload. Here is a link to the Google Colab file. Feel free to create a copy and use it next time you embark on a journey to research content on the web. ATB.
You can adjust your query limit on the ‘num’ and ‘stop’ parameters as shown here-
