What Are the Latest Scraping APIs or Services/Websites?
6 points
21 days ago
| 5 comments
| HN
I'm working on a large web scraping project and looking to avoid building custom scrapers for every site using something like BeautifulSoup.

Anyone have recommendations for up-to-date scraping APIs or platforms that work well with a variety of websites?

mike97
21 days ago
[-]
I am actually working on a scraping API for serps, currently it only supports Google, may add more search engines if it gets any traction, it isn't ready yet, but hopefully it will be in a couple of days, if you want to give it a go let me know and I can give you a subscription in exchange for some feedback.
reply
davidsojevic
21 days ago
[-]
If you're looking to scrape any of the major search engines, eBay, Yelp, and a few others, then SerpApi has a massive number of up-to-date APIs to choose from:

https://serpapi.com/search-api

Disclaimer: I work there!

reply
nreece
21 days ago
[-]
* Shameless plug * Our feed builder combines a web scraper, AI-powered content auto-detection and data transformer to generate custom JSON and RSS feeds for any public webpage.

https://newsloth.com

reply
bootstrpppin
20 days ago
[-]
We use Bright Data - so far we've found it to be pretty great, and very very cheap.

Would highly recommend

reply
gbertb
21 days ago
[-]
reply
ipaddr
21 days ago
[-]
They are powered by:

https://github.com/spider-rs/spider

For those who want to self host.

reply
jdcampolargo
21 days ago
[-]
thanks!
reply