Can’t speak for search engines specifically. But I recently had to do a project which required me to crawl the customer’s large site and index it into a vector search for RAG for a call center.
My first attempt was to use crawl it just by doing GET requests (ie same thing as using curl). That got me nowhere. I had to use headless Chrome and Playwright.
Do any modern websites work with just curl even if they don’t block it - ie without being able to run JS?
https://pypi.org/project/ddgs/
(Assuming you prefer Python.)
> and found that most of them block the use of curl
Try again, but have curl provide a user agent string from one of the real browsers. You'll likely find that the request goes through.
Can’t speak for search engines specifically. But I recently had to do a project which required me to crawl the customer’s large site and index it into a vector search for RAG for a call center.
My first attempt was to use crawl it just by doing GET requests (ie same thing as using curl). That got me nowhere. I had to use headless Chrome and Playwright.
Do any modern websites work with just curl even if they don’t block it - ie without being able to run JS?