> Why your website is invisible to the agents visiting it.
I only _wish_ I could make a website invisible to agents. Instead now we're hostages to Cloudflare /shrug
Edit: so this isn't as "driveby-ish": the tldr; seems to be, if you want your site crawl-able, don't go overboard on the JS. This was good wisdom back then just as it is now in the current situation.
I have been using llms.txt for that
I really don't know why he did this, his website has llms.txt
> Why your website is invisible to the agents visiting it.
I only _wish_ I could make a website invisible to agents. Instead now we're hostages to Cloudflare /shrug
Edit: so this isn't as "driveby-ish": the tldr; seems to be, if you want your site crawl-able, don't go overboard on the JS. This was good wisdom back then just as it is now in the current situation.
i think its better use llms.txt for this