Ask HN: How do you search the web programmatically these days?
2 points
8 hours ago
| 2 comments
| HN
For the first time in a long time, I need to query a search engine programmatically, and found that most of them block the use of curl, etc.

So, my question is simple: how do you solve the problem? I've tried searxng with mediocre success, but it seems a bit heavy to have to be running a complete separate service for this one thing that I only need every once in a while. I haven't tried using a service that requires an API key, simply because I'm not sure which direction to go or who to go with.

Just thought I would ask here first.

dserban
3 hours ago
[-]
https://pypi.org/project/ddgs/

(Assuming you prefer Python.)

reply
pwg
8 hours ago
[-]
> and found that most of them block the use of curl

Try again, but have curl provide a user agent string from one of the real browsers. You'll likely find that the request goes through.

reply