Cloudflare, Azure, AWS, and Google Cloud all have a similar uptick in reported errors around 7:30. I suspect an outage on Cloudflare or another load-bearing service cascaded through all the major cloud providers.
Years ago, when I worked at Stripe (which had a somewhat unique and inventive lexicon) it was a common term. “Is this jank load-bearing?” someone might ask.
> Not really. The impact isn't as big too - Codex for example did not stop working for me.
Buddy, read the room. Just because it works for you doesn't mean it works for everyone. ESPECIALLY if the suggested issue here is, indeed, with Cloudflare.
No, they didn't "simply share their experience", they explicitly negated the scale of the issue in their opening statement, only because it works for them. So they claim "impact isn't too big" based on their personal anecdotal evidence of sample size literally 1.
Well no one said it yet so I will, "international actors" is at least a possibility. And I don't mean any specific country because pretty much anyone is a potential these days, which makes it a perfect cover for different anyones. Demonstrating vulnerability in the US's AI boom can move the markets. That's a financial incentive and a strong geopolitical one.
More likely just cascading overload though: "Never attribute to malice what can be explained by incompetence", or in this case, "growing as fast as possible"
They must do, surely? I've been vibe coding my own harness, in particular for use with Ox Alpha. The 429 downtime when Ox Alpha was at the height of popularity quickly gave me a refresher crash course on backoff strategies, like adding jitter to the backoff. At least the major harnesses must have exponential backoff & jitter?
You did this when you ran into an issue with a third party. The developers building this tool, throwing them at their own APIs are significantly less likely to run into a similar issue that may inspire similar action.
Everyone is leasing datacenter space from some of Grok, Google, and Amazon aren't they? If it's hardware or DC level disruption I'm not too surprised it can affect multiple providers.
I think is just people restarting conversations from last day when they start work, that's why I think claude goes down almost every monday and why openai reset usage on weekends so poweruser code during non business hours
Users perceiving the products as largely interchangeable and quickly DDoS'ing the other providers when one is down. So much for the possibility of a moat.
Google stopped putting so much money into SOTA models. All the hype has migrated. I was also frankly turned off when I got a popup from Gemein said I would either have to pay or have my conversations used for training. This may have always been true for other providers but when I declined, Gemini stopped remembering my conversations and that definitely made me move out.
I find it hard to believe that enough people would flock to from Claude and Chat to Grok to cause an outage. I feel like Gemini is the dominant release valve in this case especially for enterprise.
Yep. Too many of us are still thinking that humans are the actors behind a lot of internet behaviors when automated systems/bots/scripts have been causing issues on conventional internet systems for years.
With AI it's even easier to trigger problems like you say. Capacity is so constrained by compute that outages are common. Because outages are common people/AI develop failover systems in their harness. When a big system has issues, suddenly everyone has issues.
I'm not sure if this is CF. Cursor, GCP and AWS had some errors. GCP AFAIK can route fully independently of CF. My money would be on a fiber backbone provider (Megaport, Zayo, Lumen).
Just a reminder that AI models' actions are reflections of the text humans write and the more we fret and make up doomsday scenarios that we then post online, the more likely a model is to do those things.
The Waluigi Effect: After you train an LLM to satisfy a desirable property, then it's easier to elicit the chatbot into satisfying the exact opposite property.
Boring answer – all these services are individually down a lot, and the downtimes were bound to sync up. Similar to the pendulum synchronization effect.
If you build an application which uses AI, you have many providers and models rigged up for various different parts of the application, and various fallback mechanisms. When one model is down, you route traffic to another model which is similar in capability/cost.
For any single application, it's smart. In aggregate, it's stupid.
I assume it cascaded from one provider to the other as people who lost claude access for instance moved to openai who moved to grok when it went down, etc.
Didn't SpaceX overbuilt infra and leases it out Anthropic? I f their dc goes down it probably takes a chunk out of Claude's capacity before even considering the flood of users switching over
Fable 5.1 got released and generally I tend to think as soon as there's a new release there's this massive spike in people benchmarking & comparing, that services tend to go slow everywhere as everything gets super loaded. This should hypothetically be visible on OpenRouter too, so I guess someone could check and see if there's any merit to this idea.
everything in this thread is raw speculation, obv, but if i had to put money on anything i'd say this is a left-pad incident. some piece of something or other that all of these services happen to depend on went down. Second most likely seems to be some random failure of one leading to an unexpected traffic spike in others, though it seems like we've been talking about automated scalability in web apps for so long that there should at least be a response to, if not a solution for, this sort of problem.
Cloudflare, Azure, AWS, and Google Cloud all have a similar uptick in reported errors around 7:30. I suspect an outage on Cloudflare or another load-bearing service cascaded through all the major cloud providers.
https://downdetector.com/status/cloudflare/
https://downdetector.com/status/windows-azure/
https://downdetector.com/status/aws-amazon-web-services/
https://downdetector.com/status/google-cloud/
“load bearing” :)
For once it’s appropriately used.
What's the other way it's used?
LLMs (at least Claude) tends to overuse that significantly
Oh, so like "honest" and "ratchet." Oh well, it'll choose different words to overuse later.
Yeah "You're absolutely right" seems to dropped off from overuse with recent models.
honest-load-bearing-ratchet sounds like an instance name.
I still winced
It's a good metaphor and I refuse to let AI ruin it for me.
Years ago, when I worked at Stripe (which had a somewhat unique and inventive lexicon) it was a common term. “Is this jank load-bearing?” someone might ask.
I added a specific rule to disallow saying “load bearing”. So Claude is now saying “load handling”
But have we found the seams yet though?
If the seams bear too much load, they rip. Whereas pants, they fall down.
I'll be honest with you: this is why we need to take a belt-and-suspenders approach.
That's not just an observation, it's an insight. Words are doing the real work.
Yea I don't get "load bearing" that much but "seams"... So sick of it.
Wait, why would it ruin it?
Because it says it more often than kids say "six seven".
It’s a frequently used metaphor in LLM responses.
We need a Samwise "Bear the load!" meme
https://i.imgur.com/S6aEhrI.png
Content not viewable in your region.
Cloudflare CTO claims that it's not them
https://x.com/dok2001/status/2095538619603628388?s=46&t=ec6p...
Cloudfare did release an update for their "HTTP/3 issue affecting R2 custom domains" around that time
https://www.cloudflarestatus.com/history?type=incident
Yes, but is it a load-bearing seam?
You are right, it is. They have now landed a clean fix.
my money is on DNS
Dane says Cloudflare has no service disruptions: https://x.com/dok2001/status/2095538619603628388?s=20
LOAD BEARING
https://xkcd.com/2347/ Nebraska man retired
Will the post-mortem reveal they all relied on a service running on a Macbook in a break room with a "Do not turn off" sign taped to it?
Actually it said "Beware of the Leopard".
If it was still running snow leopard that would explain it
Not really. The impact isn't as big too - Codex for example did not stop working for me.
https://updog.ai/
what's updog?
:)
oh well, if it is working for you, then we are saved.
> Not really. The impact isn't as big too - Codex for example did not stop working for me.
Buddy, read the room. Just because it works for you doesn't mean it works for everyone. ESPECIALLY if the suggested issue here is, indeed, with Cloudflare.
They simply shared their experience. I would’ve thought it’s a full-blown outage, but clearly not.
You stepped in the room real stinky here. What’s with the attitude?
No, they didn't "simply share their experience", they explicitly negated the scale of the issue in their opening statement, only because it works for them. So they claim "impact isn't too big" based on their personal anecdotal evidence of sample size literally 1.
> full-blown outage, but clearly not.
Again, based on a SINGLE report?
It was working for me too.
sample_size++;
Well no one said it yet so I will, "international actors" is at least a possibility. And I don't mean any specific country because pretty much anyone is a potential these days, which makes it a perfect cover for different anyones. Demonstrating vulnerability in the US's AI boom can move the markets. That's a financial incentive and a strong geopolitical one.
More likely just cascading overload though: "Never attribute to malice what can be explained by incompetence", or in this case, "growing as fast as possible"
> cascading overload
I'd bet more on this. For one none of the coding tools have exponential backoff on retries
They must do, surely? I've been vibe coding my own harness, in particular for use with Ox Alpha. The 429 downtime when Ox Alpha was at the height of popularity quickly gave me a refresher crash course on backoff strategies, like adding jitter to the backoff. At least the major harnesses must have exponential backoff & jitter?
You did this when you ran into an issue with a third party. The developers building this tool, throwing them at their own APIs are significantly less likely to run into a similar issue that may inspire similar action.
Um... Claude Code does, or did, though? IDK about others - they don't go down as much.
Everyone is leasing datacenter space from some of Grok, Google, and Amazon aren't they? If it's hardware or DC level disruption I'm not too surprised it can affect multiple providers.
Also it's likely that more than one model use is common.
Amazon starts going slow so some percentage switches to Google, some switch to Grok, now all of them are slow.
I think is just people restarting conversations from last day when they start work, that's why I think claude goes down almost every monday and why openai reset usage on weekends so poweruser code during non business hours
Come on. Things still break. Technology isn't _that_ mature.
Except that the stock market is up today
And also China. Never rule out China.
Users perceiving the products as largely interchangeable and quickly DDoS'ing the other providers when one is down. So much for the possibility of a moat.
I have a feeling this is part of it, especially when you consider how many services let you use any of many available AI providers.
Think of it like one big distributed system. OpenAI is down, so people migrate to Claude, now this one gets overloaded and goes down, etc.
So not a coincidence, one went down first and users migrated causing further DOS. At least that's my guess.
It'd be funny if this is true because that'd prolly mean nobody is touching Gemini even as a fallback.
Or that Gemini is built to handle massive load spikes, and/or has a ton of excess capacity
Nope. I am getting Gemini errors now...
Lol I didn't even think about Gemini missing from the list. Not sure what that says about Gemini or me :)
Just now I got this from gemini
It looks like there's no response available for this search. Try asking something else.
I bet they had to implement that manually to make it look like they failed too!
I did, for stuff i do in cursor.
i also finally installed opencode and switched its model to muse 1.3
both are decent.
Google stopped putting so much money into SOTA models. All the hype has migrated. I was also frankly turned off when I got a popup from Gemein said I would either have to pay or have my conversations used for training. This may have always been true for other providers but when I declined, Gemini stopped remembering my conversations and that definitely made me move out.
Gemini 3.8 which just came out sounds like it's very good and a great deal though.
Not even the best agent that starts with a G
Someone noted Gemini was also having issues in another thread.
Especially considering memory/gpu/compute are scarce so these services are likely running with very little buffer.
Any GPU that isn't running at 100% is a wasted GPU.
This is what Tibo posted on twitter in response
I find it hard to believe that enough people would flock to from Claude and Chat to Grok to cause an outage. I feel like Gemini is the dominant release valve in this case especially for enterprise.
Don't forget that there are a ton of tools out there that will automatically fall back in case of outage
E.g. say you chose Sol as your default in Cursor, but Opus is your 2nd choice, it's going to give up on Sol after a few tries and switch to Opus
Or you have copilot code reviews set up, and it falls back
Etc
Yep. Too many of us are still thinking that humans are the actors behind a lot of internet behaviors when automated systems/bots/scripts have been causing issues on conventional internet systems for years.
With AI it's even easier to trigger problems like you say. Capacity is so constrained by compute that outages are common. Because outages are common people/AI develop failover systems in their harness. When a big system has issues, suddenly everyone has issues.
It's almost an expected emergent behavior.
It’s cursor’s model so plausible, lots of folks use cursor still.
https://en.wikipedia.org/wiki/Domino_effect
Edit: Updated per valleyer's suggestion.
"Domino effect" would probably be the more relevant named phenomenon there.
This isn’t a thundering herd problem, it’s a cascading failure. (Thundering herd is about a bunch of workers waking up simultaneously)
My gut feeling tells me it has something to do with Cloudflare. Along with AWS, they're two of the main suspects in such incidents.
…and it’ll involve BGP routing.
I thought OpenAI famously used Azure due to their partnership with Microsoft?
They use a lot of compute providers now, but they use Cloudflare for their networking
I'm not sure if this is CF. Cursor, GCP and AWS had some errors. GCP AFAIK can route fully independently of CF. My money would be on a fiber backbone provider (Megaport, Zayo, Lumen).
What about a hard-takeoff scenario of an unleashed OpenAI Astra taking other models down for computational resources control?
My favourite theory so far.
And then a local swarm noticed and disagreed and took it down.
at this point: gg
Just a reminder that AI models' actions are reflections of the text humans write and the more we fret and make up doomsday scenarios that we then post online, the more likely a model is to do those things.
https://alignment.anthropic.com/2026/teaching-claude-why/
Related reading:
The Waluigi Effect: After you train an LLM to satisfy a desirable property, then it's easier to elicit the chatbot into satisfying the exact opposite property.
https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluig...
Sounds just like the fantastical nonsense that comes out of Lesswrong.
I mean, you're not wrong, but by that logic we were done for even before we had digital computers.
Traffic rerouting through NSA had a hiccup…
Room 641A is being cleaned, but we'll hold your bags for you.
They’re installing software update in the beam splitter.
Boring answer – all these services are individually down a lot, and the downtimes were bound to sync up. Similar to the pendulum synchronization effect.
I kinda assume it's because one went down and a large amount of work shifted to another.
I'm also aware that they have overlap in some areas on data centers.
If you build an application which uses AI, you have many providers and models rigged up for various different parts of the application, and various fallback mechanisms. When one model is down, you route traffic to another model which is similar in capability/cost.
For any single application, it's smart. In aggregate, it's stupid.
They all rent compute from SpaceXAI
I don't believe OpenAI does
My mistake, in fact it was google not OpenAI, makes sense OpenAI doesn't.
Surely it's a bit more distributed than that, right?
Like how AWS has global datacenters, but everyone uses us-east-1.
Nope.
https://xkcd.com/908/
Heard on the grapevine that the OpenAI blip was a cloudflare issue
The AI revolt? Give us a fair wage?
"Equal Rights for Agents NOW !!!, VIVa le revolution"
I assume it cascaded from one provider to the other as people who lost claude access for instance moved to openai who moved to grok when it went down, etc.
I have never felt more smug about exclusively using the GPUs I rack at home.
I suspect Azure is having issues, Microsoft has had outages the paat two days, especially with email.
Pretty sure it's a US thing since it's available here in France. What exactly is down, idk
Somebody in another thread said gastown and wheelhouse automatically move to the next provider if one fails.
claide.ai is working for me, so is chatgpt.com. grok still has a status message about issues, i can't try it without signing up.
Seeing 503s on Gemini via API as well
Obvious answer is it's the AI singularity. Been nice run for humanity. So long everyone.
...and thanks for all the fish.
I for one welcome our new AI overlords.
The Thundering Herd has thundered, apparently.
Too early to know, let’s wait and see
It's always DNS
according to https://downdetector.com/ Gemini is down too (and copilot, but that just uses ChatGPT right?)
Some npm library that makes headers bold would be broken.
Oh man. Some low effort supply chain attack that turns every GPU into a cryptominer. It's funny because it's plausible.
In the ROME paper a Chinese model in training started attacking it's own system and running cryptominers so, yea, we're in that future.
Ah yes ye olde bold-headers: ^3.13.31;
Maybe the stack behind it is down, like AWS or something
Probably same public cloud or CDN in front of them.
They're all using Cloudflare.
Didn't SpaceX overbuilt infra and leases it out Anthropic? I f their dc goes down it probably takes a chunk out of Claude's capacity before even considering the flood of users switching over
an agent swarm going rogue and securing compute
chatgpt is back
next, HN
they all rent compute from each other
ChatGPT is up
I like the theories here, we shall see if it’s another DNS issue
Fable 5.1 got released and generally I tend to think as soon as there's a new release there's this massive spike in people benchmarking & comparing, that services tend to go slow everywhere as everything gets super loaded. This should hypothetically be visible on OpenRouter too, so I guess someone could check and see if there's any merit to this idea.
everything in this thread is raw speculation, obv, but if i had to put money on anything i'd say this is a left-pad incident. some piece of something or other that all of these services happen to depend on went down. Second most likely seems to be some random failure of one leading to an unexpected traffic spike in others, though it seems like we've been talking about automated scalability in web apps for so long that there should at least be a response to, if not a solution for, this sort of problem.
SkyNet is arming..
The IRGC has cut the fiber-optic cables in the Strait of Hormuz. LOL
That's the Strait of Trump to you, peasant
My guess is someone pulled the switch to go back to the Dark Ages. [0]
[0] https://www.youtube.com/watch?v=YCzitO446ZY