URL scraper or grabber

[waca2015]

I am looking for software that will allow me to input a list of keywords and then grab first 5-10 organic URLs that show up in search engine. Everything that I found to this point is not really working that grate. Any recommendations? Thank you for your help.

[spikey_richie]

I don't know of a tool that would do this for you, but you could look at using Visual Studio Community Edition with Selenium and the Firefox Selenium IDE to get you started...

 

Good luck!

[Seahorsepip Veteran]

I would highly recommend to use the google search API instead, using a scraper/bot/whatever will not only make it harder to write the code but it will also end up breaking from time to time and it can result in your ip getting blacklisted(enter captcha before continue) for unusual traffic.

 

The google search API is mainly targeted towards searching on a specific set of domains but you can also just enter something unspecific as *.google.com as domain and set it to search the whole web in advanced settings: https://stackoverflow.com/questions/4933097/how-can-i-search-the-entire-web-from-google-custom-search

 

The API can be called from any programming language code but can also be called from a powershell script that lists the first 5 urls in a text file as example:

 

$url=https://www.googleapis.com/customsearch/v1?q=seahorsepip&cx={YOUR_ENGINE}&key={YOUR_API_KEY}
$response = Invoke-RestMethod -Uri $uri
$results = $response.items
$output =
	Foreach($item in $response.items)
	{
		write-output $item.link
	}
$output | Out-File "C:\path\to\directory\with\search\results.txt"

Above powershell script isn't tested but should output all urls found in a text file.