I built an AI assistant for our docs at Handsontable, and then Grok found it. 4.5 million edge requests and 87,000 page renders in only 4 hours, all from xAI's Colossus data center in Memphis. I caught it during the scan and pushed a fix to redirect traffic to our MCP, then spent a week analyzing the logs and learning how crawlers work, what they look for, and what they ignore.
I built an AI assistant for our docs at Handsontable, and then Grok found it. 4.5 million edge requests and 87,000 page renders in only 4 hours, all from xAI's Colossus data center in Memphis. I caught it during the scan and pushed a fix to redirect traffic to our MCP, then spent a week analyzing the logs and learning how crawlers work, what they look for, and what they ignore.
This would be interesting to read if not for the LLM prose style.
> That last detail was the tell: every crawl session was a fresh browser with no persisted thread id
Yep, ai;dr, closed tab
Another example of uselessness of llms.txt.