No big surprises here. Big outages at big firms seem to always look like this.
The electricity industry has this notion they call a black start, where they bring the grid online from zero. It's a very difficult operation because you can't start a generator straight into an offline grid otherwise it gets hit with full demand and can't keep frequency in the right range, which can damage equipment. So they have to start by disconnecting all the loads and then bringing load blocks back in sequence, trying to roughly balance available generation with the size of the next block. In practice black starts are extremely rare, but they can happen after a cascading failure in which something causes some generating stations to trip, redirecting load onto the remaining stations which can't handle it causing them to trip in turn. The entire grid can collapse in just 60 seconds or so once that kind of runaway failure starts.
This kind of outage is very analogous to a black start. A few servers get into trouble - maybe they're in a crash loop or maybe their caches are cold - the load balancers shift traffic onto the remainder which then can't handle the load either and so the failure cascades. Once a service is in this state sometimes the only remedy is to disconnect all load, bring the servers back to stability, then bring load blocks back online in small chunks, making sure to never hit anything with the full load immediately to ensure any caches or JIT compilers have time to warm up.
It's the sort of thing that requires a bit of preparation because naive ways of doing it like just randomly dropping 90% of requests will yield a broken service for everyone. Ideally your LB can drop traffic by modulus of the session cookie or similar, so users are let back in atomically.
Black starting a service is obviously quite hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster. I used to be an SRE at Google and although we regularly started up new clusters and moved traffic around between them, actually starting an isolated service instance entirely from scratch was never something we practiced. I remember in a few cases teams discovered devs had introduced circular dependencies by accident, i.e. starting service X required service X to already exist, or it depended on service Y that in turn depended on X. Sometimes you can only find these things by practicing.
I never worked in civilian power generation, but I can vouch for this on Nuclear Ships.
We had to practice load pickup on a normal start at the pier often, but we likely practiced a full SCRAM to back online (including a FULL load shutdown) more.
Having predefined lists of what gets brought up at various points helps a ton, but it's still not an easy or simple operation.
It's more testing theater on their end; it's just perfunctory rubber stamping. There was and is no real performance testing under simulated near-real world conditions.
that's not the problem with grid black start at all? black start is difficult because many types of generators themselves need power to start. it's a circular dependency problem, not a load management one.
Back in 2025, Spain did a very-dark-brown-almost-black start (with help from neighbours), starting with one hydropower plant and increasing from there.
SBB just finished their project to be able to restart the grid in 5min. (There was a famous multi hour outage a few decade ago that I think motivated this work)
The biggest surprise here is that they're using Windows for critical network infrastructure. I'm surprised a Windows process has any role here outside the Exchange servers themselves.
Not a surprise. It's Microsoft, Windows is their bread & butter for a long time. And Windows Kernel team is actually full of amazing engineers, reputable computer scientists. Disclaimer, my former boss worked at Windows Kernel team, she's the most talented engineers I have ever worked with, she then went on lead core platform engineer at Shopify many years later.
Before Linux took over the server market in 2000, Windows server was the dominant player in mid-market (enterprise market has IBM mainframes too). But you would have be old enough to remember that decades ago.
Not a network engineer so genuinely curious how these things are handled at the epic scale of 365, azure, aws and so on.
Extent of my knowledge is nginx or traefik load balancing which at this scale is virtually equal to no knowledge. But seems like there would be systems in place to simply drop a sliding percentage of excessive traffic or block some regions once critical thresholds are reached rather than compounding the problem with forwarding the growing number of requests downstream to next victim server.
They talk about their relatively low-end testing environment PASSing the July code, and being different from their high-end production environment machines, sadly hiding the concurrency bug during the test phase.
Is this commonly acceptable? I'd imagine it's a costly trade-off?
At Microsoft's scale, it's extremely difficult to simulate production scale because the sheer size of the production system is overwhelming, and many issues emerge from the combination of sheer volume and the complexity that reality tends to introduce.
Microsoft tests its stuff on their own employees (old citation, but still true today: https://devblogs.microsoft.com/oldnewthing/20110802-00/?p=10...), and that includes Exchange online. However, even though Microsoft is a pretty big company, it's still teeny compared to its customer base.
Depends on what low/high end are. If low end is 4 core and deployment is 128 cores, this is not acceptable.
But if low is 64 and high is a whole cluster, now that's a different kind of maths.
Edit: The article states the following:
> "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted."
So, it's more the former. The testing machine is probably a 8 year old server on an office shelf, and prod machines are 64-128 core cutting edge Xeons/EPYCs.
Lock contention. It was MUTEX spins in Windows 95, it's lock contention in 2026. Why this is still a thing, Microsoft?
I always this was standard to have a test setup in the production environment on the latest hardware (that’s how you qualified that hardware in the first place) running one “unit” of the system (whatever that is) at full scale and load (ideally higher).
We had this conundrum early on when we were building the Neo4j DBaaS - there was zero chance to afford to run test environments as big as production.
We solved that by sharding customers, so about every 1000 database clusters got their own k8s cluster; production then could run hundreds of those “orchestras”, but we could afford to load test one of them and be relatively confident the test was valid for production.
This relied on the various cloud vendors running the front-line load balancers that took traffic and moved it into the shards though. I was about to say that seemed to be the case for Microsoft here as well, but actually it seems they load tested an individual server with different specs than prod.
Tricky problem though - it seems they reasoned if a lower specced server passed the load test, a much beefier prod server would handle the load too. Wrong, but you can see why someone made that assumption.
Testing at production scale is extremely difficult. It’s also financially expensive. In my experience, you balance the two where big changes are tested with some simulation setup to gain confidence. Though, the ci/cd pipeline never has close to full scale automated testing.
...and many times you test to experience, and experience cannot possibly build interferences that simulate the instances you have not experienced nor what has been imagined based on that experience.
some ways of precipitating the demons that are in the ontology of not knowing what you don't know are monte carlo, or perhaps noise injection, but those are time consuming, quite wasteful, and truly hated by people used to working with deterministic machines. that makes finding corner cases very difficult
plus, mistakes are the essence of learning and sometimes mistakes are necessary.
likewise, we are a very efficient world and efficiency hates redundancy, and hates nonconformity...of which those are two of the features that make exploits widely successful and failures massively impactful
> The engineers who worked feverishly to restore services deserve commendation and will hopefully never have to buy their own beverages in Redmond again (or at least escape the next round of firings).
How long until Microsoft fires key people to the point that outages become more common? All in the name of AI, which I don't believe for a second, they're just trying to maintain stock market profitability math by firing enough, they'll just hire more people when they make more.
The opportunity/command to exit is not always negative. In many of my career moves, it turned out that the next job was better and came far sooner than the severance calculation had allowed for. Some within weeks. Hardly had time to relax. It seems like for talented individuals opportunity always pops up.
> How long until Microsoft fires key people to the point that outages become more common?
Already happening. Granted, not a global outage like this one, but there's not a single day without a service degradation or advisory on the admin portal for 365. The "It's now called Microsoft 360" memes are well earned, and their support is atrocious. I get the feeling that everyone who knows how it works is gone.
Nurses reported many hospitals paid significant bonuses and hazard pay during COVID, including (already mandated) time-and-a-half, as nurses were in short supply. Many nurses were being paid at hourly rates above that of senior software engineers.
That's not to say their job was or is easy: nursing is far more hazardous than SWE, and significantly more physically and mentally demanding, too. But I do think many got more compensation than your $5 gift certificate implies.
But as a software engineer, if I work overtime on an outage like the above, I'll get precisely $0 of additional pay for it.
Valid only upon a qualifying transaction with a minimum purchase amount of fifty dollars ($50.00). Subject to additional terms, conditions, and restrictions. The display, wear, or possession of any "Hero" sticker shall not be construed, interpreted, or implied as an official or unofficial representation, certification, or endorsement that the wearer possesses the status, title, or attributes of a "hero".
In sysadmin speak this is hoping that Microsoft gives them free beverage credits (or give them some kind of bonus because of their efforts) and don't fire them as soon as the fix is deployed to scapegoat them.
In a rare occurrence, The Register sides with the Microsoft developers. Not that they're anti-developers, but they are anti Microsoft because of its well-earned reputation for EEE and Halloween Documents.
It means they deserve "Thanks" (buying a beer/coffee is a token Thank You) and hopefully they are removed from the pool of candidates for the next layoff.
Surprised so many people are struggling with this...
Having been on incidents like this before though, yea, that aint happening.
> Having been on incidents like this before though, yea, that aint happening.
Been there, through that, and sadly you're right. The bitter part is I didn't want extra compensation or anything. Just a thank you and a day off is more than enough for me, so to recharge and ride the wave of what we (not I, we) have accomplished, so I can work with extra motivation for some time.
Instead, got a 5 minute coffee break and dived to the next non-urgent ticket. Yeah, great way to thank and motivate the people who spent 24 hours in an underground office cutoff from the world, trying to save a system from melting down.
> Having been on incidents like this before though, yea, that aint happening.
Yeah, instead what will happen is their manager will ask them why they didn't get as many story points done this sprint as they did last sprint and their performance will be marked "degraded".
No, no, i think we can and should still load all the eggs into a single basket...we simply have to do so at random frequencies and speeds in order to avoid cracking the eggs, plus when contention inevitably happens, we ask that the egg loaders simply hold off a little...to avoid such traffic, and until egg loading can resume! /s
>> "the processes that led to an update with a latent defect reaching production deserve a long, hard look. "
Indeed they do deserve a long hard look.
Toss1's conjecture:
We are now seeing the beginning of an era of latent technical debt brought about by too rapidly firing developers and switching to AI code generation (nevermind vibe-coding). Even well-prompted and well-monitored by top expert coders and architects, there is latent technical debt, and it is more diffuse, harder to find, and harder to fix, and will start showing up in outages like this.
(It first seemed the ATC outage was another example, but it now seems it was merely the usual management incompetence if not an outright manipulative ploy to 'justify' more spending for a single-bid contract [0])
"Complex systems are kept stable by organisational knowledge of their weaknesses and how to compensate for them. Organisational change erodes this knowledge while the underlying weaknesses remain. A previously stable system thus becomes increasingly vulnerable to failure."
When it is said that software is a solved problem, what is ment by that is really that programming, typing up the code, is a solved problem. Software engineering, systems architecture and the complexities in distributed systems are not, while leads to confusion when not being mindful of the difference.
Interesting that it's their EVP of Copilot, Agents, and Platform at Microsoft, building the next generation of AI-powered tools for work who spoke up for he post-mortem instead of their CTO.
Does that mean Microsoft's internal code are 100% outsourced to agents / AI and human review just won't be able to catch latent bugs like this?
This is what happens when you burn billions for no reason.
Github copilot had INSANE usage. Cursor switched away from per request plans in 2025, Copilot did it this year with all the reasoning and more expensive models. It was insane, you could write out a whole page of features, and have it run on one request for hours depending on the loopholes you exploited (some models were discounted for some time, basically free in some cases, and shenanigans around switching in betwen agents)
We talk about opus et. al being heavily subsidized at like 1-2k for $30-$100. I remember seeing figures in the 10s of thousands in subsidizations for GHC. There's no way they weren't losing just massive piles of money.
Doesn't the article state the opposite? The low end was not the problem. It was high end that was the problem.
I like ragging on Microsoft as much as the next guy, but honestly this problem sounds complex & hard to predict. AI or not, this feels like the type of bug that would have slipped by me as well.
The response clearly could have been stronger though.
Seems like the issue happens readily at high load on a system with many cores. Once the conditions were right it seemed to impact a large proportion of their fleet. They just didnt have that combination in their test/validation pipeline. They likely have some sort of load test in their pipeline, but it's running on machine that's much smaller than their production machines... so it didn't trigger the lock contention bug.
The opposite is true running in production for the last several months. They were runnign on big production machines with enough cores, but never quite had enough load to trigger the lock contention. But once both conditions were true, big machines and high load, due to some unrelated failover elsewhere suddently bringing more load, we then had the magic combination.
It's really not unreasonable to expect an enormous hyperscaler like Microsoft to do load testing on hardware that exactly matches their production systems.
I read "more than one low end" as "more testing environments than one low end" rather than "more low end environments than one". It's a snarky way of saying they should have been testing high end environments for no more reason than they had the money to do it, rather than saying they should have been testing the low end more.
False. A lot of working people, probably even you, are tangentally invested into these stocks through indexes, pensions, endowments, and retirement accounts.
What's false about it? I didn't say working people don't own stocks. They just don't own most of them. Half of the benefit of any increase in stock prices goes to 1% of the people.
Workers benefit much more from higher wages than higher stock prices. Simply because a significantly higher proportion of their income comes from wages.
Whenever someone says "Think about the retirees and other ordinary shareholders" in response to a suggestion to pay workers more than absolutely necessary, they're full of shit.
I don't care who owns most of them, I care that a big part of my retirement savings are in stocks. It may be an inconvenient fact for some people, but my interests are aligned with "the rich" for this particular thing. If some rich guys get richer when MSFT does well, but I also get to have a more comfortable retirement as well, then that's fine by me.
This "rich people will benefit so it's automatically bad" mentality is counterproductive and borne out of jealousy.
Your interests are only fully aligned when you're actually retired and 100% of your income comes from stock investments.
If part of your retirement income comes from rental property, workers being paid more is better for you because they can afford higher rents.
Ordinary workers would better off if they were simply paid more and could put more money into their retirement savings. There's a crossover point where investment gains outpace contributions but it takes most workers a very long time to hit it. Front-loading retirement saving and letting compound interest do its magic over a long time is the way to go. And front-loading is only possible when workers get paid good money early in their careers.
This "whatever's good for rich people is equally good for me" is a mentality born out of being bad at math, and assuming that everyone who disagrees is a raging communist (sorry).
Capitalism is great, moar stock gains are great - I enjoy both. But it's important to look at real numbers.
Did I miss something? I don't see where someone in this comment chain argued for workers to not be paid more. I very much enjoyed being paid well so that I could retire early! It would be great if others get to be similarly fortunate.
All me and the other guy are saying is that it's possible to not be filthy rich and still want stocks to do well. "Think of the poor starving shareholders" implies that we commoners shouldn't care or want that. But that's false. I'm no longer working, and have no rental income, so I need my investments to do well.
This is HN, 99% are white collar workers like me invested into pensions, investment accounts, funds, etc. that are mostly stock backed due to working at companies with these programs. Even government and non-profits are the same.
We all need to side with whoever makes the charts go up and to right to ensure our post-retirement lives are well funded.
Myself and many like me don't care what side that is and we don't care much for policies, groups, or programs that are counter to that.
My guy what's going to make your retirement account go up faster? Contributions or gains?
For at least the first 11 years of a career, contributions outweigh gains. And only if you continuously contribute the same amount every month.[1] But in fact most people's contributions increase because their salary increases, so the actual crossover point might much later, such as year 20.
Being paid more, especially early career, is just more useful to the 99% white collar workers on HN. It lets you save more money for retirement. Compounding more money for longer gives you a better retirement. This is simple math. It isn't a class conflict argument.
The mantra of "shareholder primacy" sounds great in theory but just becomes an excuse for short term profit boosts for a small class of shareholders usually at the expensive of the long term health and growth of the company and long term value seeking shareholders.
Sort of similar to how most countries with "people's" or "democratic" in the government's name tend to be democratic only for the ruling party.
In this case ensuring adequate engineering resources for possibly the most widely used email system in the world doesn't seem crazy. How many long term customers will they loose because of this incident? Perhaps they did have enough resources, but given the recent layoffs to chase AI I sort of doubt it.
Not for me and probably a lot of people and institutions too. We have retirement accounts, endowments, and pensions tied to index funds and the index funds have these stocks in them.
We need these companies to do well to ensure we retire and grow well.
Who cares? For most working people like myself invested into these companies through funds, pensions, investment accounts, savings, etc., all we care about is up and to the right forever or at least when we come into retirement.
Pretty interesting that testing was aimed the worry: low spec boxes, and missed a different problem: concurrency contention for high spec boxes. Gotta be covered by someone's rule of testing, I imagine. Also, woof.
1. They did not have proper cell-based architecture to partition their system into failure domains ("In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure.").
2. They did not have proper monitoring to see the encroaching collapse.
3. Stuff just keeps falling off: "Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day"
These kinds of incidents should inject healthy dose of techno-skepticism in the larger conversation. The world is teetering on the edge of turning over all our processes to LLMs and retiring human labor for good.
But at the end of the day these are servers strung together sending packets over copper wires. They are extremely reliant till they are not.
When crisis strikes you can only hope there is a squad of well trained humans who can step in to untangle these messes. But as more of the labor force is stripped out of responsibilities, the longer these outages will take to resolve. Imagine being jolted out of your year-long UBI-funded vacation to login and resolve a multi-continent server outage that you only have a vague memory of.
From what I've observed, SV is doing anything and everything it can to make it reality. It's scary to see how successful their campaign has been so far.
Also, don't forget that e-mail is a distributed system and servers retrying always keep a pressure on you, despite their best intentions, and spaced retries.
So, even when you fix the issue, it takes a bit to clear up, and the load gauge stays at the red zone for a while.
Anytime you run a proxy system for a large number of clients you don’t control you need a way to cold start. This means being able to allow traffic through (cheaply!) on a client basis, rejecting the rest.
This lets you start at a very small amount of load and gradually ramp up. But you can’t reject randomly or goodput stays at zero and the backlog never decreases.
I believe SDN systems can do that with ease, but what would be the equivalent method on older systems? Selectively allowing IPs to go through the trickle load into the system?
>> "an update that returned the code path to the previously known good state in Windows."
I know I shouldn't be surprised, but I still struggle to imagine such high-capacity servers running with windows. Part of me wonders why any important server would be running windows. And I do chuckle at the "update" that returned windows to a previous state. Never admit a rollback. Never admit a previous update was less than perfect. Hold the line MS. Hold that line.
But if they wouldn't do it, Windows will be even worse.
Moreover, my current switch to Linux shows that Linux is not so much better in stability as it is in public imagination. Once a month or two, I have KDE freeze so hard I barely can recover without reset (sometimes can't). Still can't find the reason. Hardware seems to be okay, it is an expensive gaming laptop.
Also, out of RAM can lead to GUI crashing even with swap enabled. True, this is unrelated to servers, which run without GUI, but still weird that it happens.
Also, when my laptop runs for weeks without reboot, it has issues with VRAM being occupied and not freed for some reason, which causes game crashes. With all the talk about Linux infinite uptime I expected better.
However, I almost can't complain about my homelab server. If there is enough disk space, it can run for months. Unfortunately, lately there are so many patches, I need to reboot almost each day (at least podman-hosting VM, if not proxmox host)
For the last half a year I run my homelab, I only had one suspicious shutdown, for which I am not sure I understand the reason well.
Alot of 365 is using Exchange/Sharepoint/Skype for Business code bases which were Windows specific. With Exchange, there was specifically role called "Client Access" which clients connect, the system looks up which backend server has your mailbox and proxies your connection there. Reading between the lines, my guess is this system was reused for Sharepoint/Teams/Office and they could swing off of it to secondaries but Exchange is hard locked on it.
Microsoft does greenfield new stuff on Linux mostly.
Yeah it's pretty crazy that Microsoft, a company most well known for it's Windows operating system, would use that same operating system on its servers.
That is not a Linux server running MS services. That is a Linux server developed by Microsoft for customers who want a Linux server for their own service infrastructure
This MS rando on X just leaking his 'abridged' version instead of letting the official account post the report/blog later or whatever? Gotta be 'first' right bro, and now the source of every rabid tech blog for the rest of the day. bah
No big surprises here. Big outages at big firms seem to always look like this.
The electricity industry has this notion they call a black start, where they bring the grid online from zero. It's a very difficult operation because you can't start a generator straight into an offline grid otherwise it gets hit with full demand and can't keep frequency in the right range, which can damage equipment. So they have to start by disconnecting all the loads and then bringing load blocks back in sequence, trying to roughly balance available generation with the size of the next block. In practice black starts are extremely rare, but they can happen after a cascading failure in which something causes some generating stations to trip, redirecting load onto the remaining stations which can't handle it causing them to trip in turn. The entire grid can collapse in just 60 seconds or so once that kind of runaway failure starts.
This kind of outage is very analogous to a black start. A few servers get into trouble - maybe they're in a crash loop or maybe their caches are cold - the load balancers shift traffic onto the remainder which then can't handle the load either and so the failure cascades. Once a service is in this state sometimes the only remedy is to disconnect all load, bring the servers back to stability, then bring load blocks back online in small chunks, making sure to never hit anything with the full load immediately to ensure any caches or JIT compilers have time to warm up.
It's the sort of thing that requires a bit of preparation because naive ways of doing it like just randomly dropping 90% of requests will yield a broken service for everyone. Ideally your LB can drop traffic by modulus of the session cookie or similar, so users are let back in atomically.
Black starting a service is obviously quite hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster. I used to be an SRE at Google and although we regularly started up new clusters and moved traffic around between them, actually starting an isolated service instance entirely from scratch was never something we practiced. I remember in a few cases teams discovered devs had introduced circular dependencies by accident, i.e. starting service X required service X to already exist, or it depended on service Y that in turn depended on X. Sometimes you can only find these things by practicing.
I never worked in civilian power generation, but I can vouch for this on Nuclear Ships.
We had to practice load pickup on a normal start at the pier often, but we likely practiced a full SCRAM to back online (including a FULL load shutdown) more.
Having predefined lists of what gets brought up at various points helps a ton, but it's still not an easy or simple operation.
It's more testing theater on their end; it's just perfunctory rubber stamping. There was and is no real performance testing under simulated near-real world conditions.
The biggest challenge of bringing a resilient system back online after an outage, is the resiliency of said system fighting back.
that's not the problem with grid black start at all? black start is difficult because many types of generators themselves need power to start. it's a circular dependency problem, not a load management one.
Back in 2025, Spain did a very-dark-brown-almost-black start (with help from neighbours), starting with one hydropower plant and increasing from there.
It went remarkably well.
Somewhat related: https://company.sbb.ch/en/railway-development/services-rus/e...
SBB just finished their project to be able to restart the grid in 5min. (There was a famous multi hour outage a few decade ago that I think motivated this work)
3-5 days in the UK according to "regulations". Some talk of it potentially taking 7-14 days under certain bad circumstances.
Happened to Spain last year. The black start.
In software servers this is called the “thundering herd” problem. It is why clients are coded do not just exponential backoff but random offsets too.
It's also a well known problem, so no excuse for that.
a well known problem with more than a few known fixes, one of which is load shedding your upstream based on protocol aware load balancrs.
I agree: no excuse.
Well stated.
The biggest surprise here is that they're using Windows for critical network infrastructure. I'm surprised a Windows process has any role here outside the Exchange servers themselves.
Not a surprise. It's Microsoft, Windows is their bread & butter for a long time. And Windows Kernel team is actually full of amazing engineers, reputable computer scientists. Disclaimer, my former boss worked at Windows Kernel team, she's the most talented engineers I have ever worked with, she then went on lead core platform engineer at Shopify many years later.
Before Linux took over the server market in 2000, Windows server was the dominant player in mid-market (enterprise market has IBM mainframes too). But you would have be old enough to remember that decades ago.
Someone has to do the testing, and Microsoft tests in production.
In 2014 Microsoft cancelled the dedicated SDET (Software Development Engineer in Test) role, triggering significant layoffs.
Just be grateful they no longer put the configuration in SharePoint. (In an Excel document).
Not a network engineer so genuinely curious how these things are handled at the epic scale of 365, azure, aws and so on.
Extent of my knowledge is nginx or traefik load balancing which at this scale is virtually equal to no knowledge. But seems like there would be systems in place to simply drop a sliding percentage of excessive traffic or block some regions once critical thresholds are reached rather than compounding the problem with forwarding the growing number of requests downstream to next victim server.
Original source information: https://x.com/clamanna/status/2122209697226948729
Better summary: https://www.neowin.net/news/microsoft-finally-explains-why-o...
They talk about their relatively low-end testing environment PASSing the July code, and being different from their high-end production environment machines, sadly hiding the concurrency bug during the test phase.
Is this commonly acceptable? I'd imagine it's a costly trade-off?
At Microsoft's scale, it's extremely difficult to simulate production scale because the sheer size of the production system is overwhelming, and many issues emerge from the combination of sheer volume and the complexity that reality tends to introduce.
Microsoft tests its stuff on their own employees (old citation, but still true today: https://devblogs.microsoft.com/oldnewthing/20110802-00/?p=10...), and that includes Exchange online. However, even though Microsoft is a pretty big company, it's still teeny compared to its customer base.
Depends on what low/high end are. If low end is 4 core and deployment is 128 cores, this is not acceptable.
But if low is 64 and high is a whole cluster, now that's a different kind of maths.
Edit: The article states the following:
> "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted."
So, it's more the former. The testing machine is probably a 8 year old server on an office shelf, and prod machines are 64-128 core cutting edge Xeons/EPYCs.
Lock contention. It was MUTEX spins in Windows 95, it's lock contention in 2026. Why this is still a thing, Microsoft?
(Man, I felt old while writing this.)
I always this was standard to have a test setup in the production environment on the latest hardware (that’s how you qualified that hardware in the first place) running one “unit” of the system (whatever that is) at full scale and load (ideally higher).
We had this conundrum early on when we were building the Neo4j DBaaS - there was zero chance to afford to run test environments as big as production.
We solved that by sharding customers, so about every 1000 database clusters got their own k8s cluster; production then could run hundreds of those “orchestras”, but we could afford to load test one of them and be relatively confident the test was valid for production.
This relied on the various cloud vendors running the front-line load balancers that took traffic and moved it into the shards though. I was about to say that seemed to be the case for Microsoft here as well, but actually it seems they load tested an individual server with different specs than prod.
Tricky problem though - it seems they reasoned if a lower specced server passed the load test, a much beefier prod server would handle the load too. Wrong, but you can see why someone made that assumption.
Testing at production scale is extremely difficult. It’s also financially expensive. In my experience, you balance the two where big changes are tested with some simulation setup to gain confidence. Though, the ci/cd pipeline never has close to full scale automated testing.
...and many times you test to experience, and experience cannot possibly build interferences that simulate the instances you have not experienced nor what has been imagined based on that experience. some ways of precipitating the demons that are in the ontology of not knowing what you don't know are monte carlo, or perhaps noise injection, but those are time consuming, quite wasteful, and truly hated by people used to working with deterministic machines. that makes finding corner cases very difficult
plus, mistakes are the essence of learning and sometimes mistakes are necessary.
likewise, we are a very efficient world and efficiency hates redundancy, and hates nonconformity...of which those are two of the features that make exploits widely successful and failures massively impactful
I think MS could afford a bit of a test lab?
Yes but something that works fine in a test lab won't necessarily cope when confronted with hundreds of millions of users.
Seems like a pretty standard big company severe outage to me. These things happen!
> The engineers who worked feverishly to restore services deserve commendation and will hopefully never have to buy their own beverages in Redmond again (or at least escape the next round of firings).
What does this mean? I didn't understand
People should buy them drink, as for the firings side note Microsoft recently had a "voluntary exit" offer mailed to a fourth of their workforce:
https://www.geekwire.com/2026/microsoft-makes-new-voluntary-...
How long until Microsoft fires key people to the point that outages become more common? All in the name of AI, which I don't believe for a second, they're just trying to maintain stock market profitability math by firing enough, they'll just hire more people when they make more.
The opportunity/command to exit is not always negative. In many of my career moves, it turned out that the next job was better and came far sooner than the severance calculation had allowed for. Some within weeks. Hardly had time to relax. It seems like for talented individuals opportunity always pops up.
I'm saying its negative for those "left behind" not so much those who took it.
> How long until Microsoft fires key people to the point that outages become more common?
Already happening. Granted, not a global outage like this one, but there's not a single day without a service degradation or advisory on the admin portal for 365. The "It's now called Microsoft 360" memes are well earned, and their support is atrocious. I get the feeling that everyone who knows how it works is gone.
I think it means that people will hopefully buy them beers as thanks
An artistic way of saying the engineers should get praised and rewarded.
You mean like the Covid nurses? With stickers that say 'You Rock!!' or 'Hero' and a $5 gift certificate to Dairy Queen?
Nurses reported many hospitals paid significant bonuses and hazard pay during COVID, including (already mandated) time-and-a-half, as nurses were in short supply. Many nurses were being paid at hourly rates above that of senior software engineers.
That's not to say their job was or is easy: nursing is far more hazardous than SWE, and significantly more physically and mentally demanding, too. But I do think many got more compensation than your $5 gift certificate implies.
But as a software engineer, if I work overtime on an outage like the above, I'll get precisely $0 of additional pay for it.
> and a $5 gift certificate to Dairy Queen?
Valid only upon a qualifying transaction with a minimum purchase amount of fifty dollars ($50.00). Subject to additional terms, conditions, and restrictions. The display, wear, or possession of any "Hero" sticker shall not be construed, interpreted, or implied as an official or unofficial representation, certification, or endorsement that the wearer possesses the status, title, or attributes of a "hero".
In sysadmin speak this is hoping that Microsoft gives them free beverage credits (or give them some kind of bonus because of their efforts) and don't fire them as soon as the fix is deployed to scapegoat them.
In a rare occurrence, The Register sides with the Microsoft developers. Not that they're anti-developers, but they are anti Microsoft because of its well-earned reputation for EEE and Halloween Documents.
It means they deserve "Thanks" (buying a beer/coffee is a token Thank You) and hopefully they are removed from the pool of candidates for the next layoff.
Surprised so many people are struggling with this...
Having been on incidents like this before though, yea, that aint happening.
> Having been on incidents like this before though, yea, that aint happening.
Been there, through that, and sadly you're right. The bitter part is I didn't want extra compensation or anything. Just a thank you and a day off is more than enough for me, so to recharge and ride the wave of what we (not I, we) have accomplished, so I can work with extra motivation for some time.
Instead, got a 5 minute coffee break and dived to the next non-urgent ticket. Yeah, great way to thank and motivate the people who spent 24 hours in an underground office cutoff from the world, trying to save a system from melting down.
Glad I'm not there anymore.
> Having been on incidents like this before though, yea, that aint happening.
Yeah, instead what will happen is their manager will ask them why they didn't get as many story points done this sprint as they did last sprint and their performance will be marked "degraded".
It kind of feels like they are saying, look, we know we caused the problem, but we fixed it. So we're good, right?
So maybe we shouldn’t be putting all our eggs in one basket?
Multiple eggs AND multiple baskets? In this economy?
Hmm, the best I can do is offer you a larger basket
Subscribe now to my egg-as-a-service product
No, no, i think we can and should still load all the eggs into a single basket...we simply have to do so at random frequencies and speeds in order to avoid cracking the eggs, plus when contention inevitably happens, we ask that the egg loaders simply hold off a little...to avoid such traffic, and until egg loading can resume! /s
>> "the processes that led to an update with a latent defect reaching production deserve a long, hard look. "
Indeed they do deserve a long hard look.
Toss1's conjecture:
We are now seeing the beginning of an era of latent technical debt brought about by too rapidly firing developers and switching to AI code generation (nevermind vibe-coding). Even well-prompted and well-monitored by top expert coders and architects, there is latent technical debt, and it is more diffuse, harder to find, and harder to fix, and will start showing up in outages like this.
(It first seemed the ATC outage was another example, but it now seems it was merely the usual management incompetence if not an outright manipulative ploy to 'justify' more spending for a single-bid contract [0])
[0] https://ajsignalnotnoise.substack.com/p/faa-air-traffic-cont...
Suggested reformulation of the conjecture:
"Complex systems are kept stable by organisational knowledge of their weaknesses and how to compensate for them. Organisational change erodes this knowledge while the underlying weaknesses remain. A previously stable system thus becomes increasingly vulnerable to failure."
When it is said that software is a solved problem, what is ment by that is really that programming, typing up the code, is a solved problem. Software engineering, systems architecture and the complexities in distributed systems are not, while leads to confusion when not being mindful of the difference.
what role did ai play in all of this?
can we blame it for the bug? can we credit it for the fix?
might be the most important part of the full report
Interesting that it's their EVP of Copilot, Agents, and Platform at Microsoft, building the next generation of AI-powered tools for work who spoke up for he post-mortem instead of their CTO.
Does that mean Microsoft's internal code are 100% outsourced to agents / AI and human review just won't be able to catch latent bugs like this?
Poor Microsoft can't afford more than one low-end testing environment.
This is what happens when you burn billions for no reason.
Github copilot had INSANE usage. Cursor switched away from per request plans in 2025, Copilot did it this year with all the reasoning and more expensive models. It was insane, you could write out a whole page of features, and have it run on one request for hours depending on the loopholes you exploited (some models were discounted for some time, basically free in some cases, and shenanigans around switching in betwen agents)
We talk about opus et. al being heavily subsidized at like 1-2k for $30-$100. I remember seeing figures in the 10s of thousands in subsidizations for GHC. There's no way they weren't losing just massive piles of money.
Doesn't the article state the opposite? The low end was not the problem. It was high end that was the problem.
I like ragging on Microsoft as much as the next guy, but honestly this problem sounds complex & hard to predict. AI or not, this feels like the type of bug that would have slipped by me as well.
The response clearly could have been stronger though.
The point they were making is that had Microsoft used high-end equipment in testing they may have found this issue before it hit production.
Considering the bug was latent for several months in production before they ran into it, unlikely they would have caught it in a contrived test
Seems like the issue happens readily at high load on a system with many cores. Once the conditions were right it seemed to impact a large proportion of their fleet. They just didnt have that combination in their test/validation pipeline. They likely have some sort of load test in their pipeline, but it's running on machine that's much smaller than their production machines... so it didn't trigger the lock contention bug.
The opposite is true running in production for the last several months. They were runnign on big production machines with enough cores, but never quite had enough load to trigger the lock contention. But once both conditions were true, big machines and high load, due to some unrelated failover elsewhere suddently bringing more load, we then had the magic combination.
It's really not unreasonable to expect an enormous hyperscaler like Microsoft to do load testing on hardware that exactly matches their production systems.
Not unless it was also under high end load
I read "more than one low end" as "more testing environments than one low end" rather than "more low end environments than one". It's a snarky way of saying they should have been testing high end environments for no more reason than they had the money to do it, rather than saying they should have been testing the low end more.
Think of the poor starving shareholders and CEO pay packages.
> poor starving shareholders
You mean you and me? You realize pensions, institutions, and individual investors are shareholders as well?
The top 1% wealthiest own 50% of all public stock. https://www.fool.com/research/how-many-americans-own-stock/ So it's mostly not you and me at all.
False. A lot of working people, probably even you, are tangentally invested into these stocks through indexes, pensions, endowments, and retirement accounts.
What's false about it? I didn't say working people don't own stocks. They just don't own most of them. Half of the benefit of any increase in stock prices goes to 1% of the people.
Workers benefit much more from higher wages than higher stock prices. Simply because a significantly higher proportion of their income comes from wages.
Whenever someone says "Think about the retirees and other ordinary shareholders" in response to a suggestion to pay workers more than absolutely necessary, they're full of shit.
I don't care who owns most of them, I care that a big part of my retirement savings are in stocks. It may be an inconvenient fact for some people, but my interests are aligned with "the rich" for this particular thing. If some rich guys get richer when MSFT does well, but I also get to have a more comfortable retirement as well, then that's fine by me.
This "rich people will benefit so it's automatically bad" mentality is counterproductive and borne out of jealousy.
This person gets it. A+++
Your interests are only fully aligned when you're actually retired and 100% of your income comes from stock investments.
If part of your retirement income comes from rental property, workers being paid more is better for you because they can afford higher rents.
Ordinary workers would better off if they were simply paid more and could put more money into their retirement savings. There's a crossover point where investment gains outpace contributions but it takes most workers a very long time to hit it. Front-loading retirement saving and letting compound interest do its magic over a long time is the way to go. And front-loading is only possible when workers get paid good money early in their careers.
This "whatever's good for rich people is equally good for me" is a mentality born out of being bad at math, and assuming that everyone who disagrees is a raging communist (sorry).
Capitalism is great, moar stock gains are great - I enjoy both. But it's important to look at real numbers.
Did I miss something? I don't see where someone in this comment chain argued for workers to not be paid more. I very much enjoyed being paid well so that I could retire early! It would be great if others get to be similarly fortunate.
All me and the other guy are saying is that it's possible to not be filthy rich and still want stocks to do well. "Think of the poor starving shareholders" implies that we commoners shouldn't care or want that. But that's false. I'm no longer working, and have no rental income, so I need my investments to do well.
This is HN, 99% are white collar workers like me invested into pensions, investment accounts, funds, etc. that are mostly stock backed due to working at companies with these programs. Even government and non-profits are the same.
We all need to side with whoever makes the charts go up and to right to ensure our post-retirement lives are well funded.
Myself and many like me don't care what side that is and we don't care much for policies, groups, or programs that are counter to that.
My guy what's going to make your retirement account go up faster? Contributions or gains?
For at least the first 11 years of a career, contributions outweigh gains. And only if you continuously contribute the same amount every month.[1] But in fact most people's contributions increase because their salary increases, so the actual crossover point might much later, such as year 20.
Being paid more, especially early career, is just more useful to the 99% white collar workers on HN. It lets you save more money for retirement. Compounding more money for longer gives you a better retirement. This is simple math. It isn't a class conflict argument.
[1] https://finance.yahoo.com/news/crossover-point-compounding-m...
It's a rhetorical jab of course.
The mantra of "shareholder primacy" sounds great in theory but just becomes an excuse for short term profit boosts for a small class of shareholders usually at the expensive of the long term health and growth of the company and long term value seeking shareholders.
Sort of similar to how most countries with "people's" or "democratic" in the government's name tend to be democratic only for the ruling party.
In this case ensuring adequate engineering resources for possibly the most widely used email system in the world doesn't seem crazy. How many long term customers will they loose because of this incident? Perhaps they did have enough resources, but given the recent layoffs to chase AI I sort of doubt it.
> It's a rhetorical jab of course.
Not for me and probably a lot of people and institutions too. We have retirement accounts, endowments, and pensions tied to index funds and the index funds have these stocks in them.
We need these companies to do well to ensure we retire and grow well.
Only about 9.5% of Microsoft shares are held directly or indirectly by people with a net worth of $1 million or less.
Who cares? For most working people like myself invested into these companies through funds, pensions, investment accounts, savings, etc., all we care about is up and to the right forever or at least when we come into retirement.
This happens when you are trying to do weird things with emails rather than just deliver them to destination.
Also this is karma because Microsoft blocked in ignorance entire datacenter networks making SMBs unable to deliver messages to their networks.
Ta-ta!
Pretty interesting that testing was aimed the worry: low spec boxes, and missed a different problem: concurrency contention for high spec boxes. Gotta be covered by someone's rule of testing, I imagine. Also, woof.
They've done this before! https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and...
Nothing more cuts and more AI won't fix!
So apparently this is the report: https://www.365fanatic.com/wp-content/uploads/2026/11/PostIn... I propose a "latent bug" drinking game.
So to recap:
1. They did not have proper cell-based architecture to partition their system into failure domains ("In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure.").
2. They did not have proper monitoring to see the encroaching collapse.
3. Stuff just keeps falling off: "Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day"
These kinds of incidents should inject healthy dose of techno-skepticism in the larger conversation. The world is teetering on the edge of turning over all our processes to LLMs and retiring human labor for good.
But at the end of the day these are servers strung together sending packets over copper wires. They are extremely reliant till they are not.
When crisis strikes you can only hope there is a squad of well trained humans who can step in to untangle these messes. But as more of the labor force is stripped out of responsibilities, the longer these outages will take to resolve. Imagine being jolted out of your year-long UBI-funded vacation to login and resolve a multi-continent server outage that you only have a vague memory of.
> The world is teetering on the edge of turning over all our processes to LLMs and retiring human labor for good.
That’s Silicon Valley fantasy, it’s not even close to reality
From what I've observed, SV is doing anything and everything it can to make it reality. It's scary to see how successful their campaign has been so far.
- we added a bug (yes, everyone does).
- our testing didn't cover it (again, this is expected occasionally).
- this cascaded to take down more than a billion people's service (WTF; this is not how you design systems).
- it took 15 hours to deploy the fix (not good, but at this scale maybe).
Feels like they are completely missing the point of where the problem is and how to fix it?
In terms of cascading failure and inability to rollback or fix quickly, MS has always been terrible.
Also, don't forget that e-mail is a distributed system and servers retrying always keep a pressure on you, despite their best intentions, and spaced retries.
So, even when you fix the issue, it takes a bit to clear up, and the load gauge stays at the red zone for a while.
Anytime you run a proxy system for a large number of clients you don’t control you need a way to cold start. This means being able to allow traffic through (cheaply!) on a client basis, rejecting the rest.
This lets you start at a very small amount of load and gradually ramp up. But you can’t reject randomly or goodput stays at zero and the backlog never decreases.
Sounds reasonable.
I believe SDN systems can do that with ease, but what would be the equivalent method on older systems? Selectively allowing IPs to go through the trickle load into the system?
IP is often hard on the proxy/load balancer, you get a really large list, which becomes expensive.
This is also what I thought, so why I asked, actually.
>> "an update that returned the code path to the previously known good state in Windows."
I know I shouldn't be surprised, but I still struggle to imagine such high-capacity servers running with windows. Part of me wonders why any important server would be running windows. And I do chuckle at the "update" that returned windows to a previous state. Never admit a rollback. Never admit a previous update was less than perfect. Hold the line MS. Hold that line.
But if they wouldn't do it, Windows will be even worse.
Moreover, my current switch to Linux shows that Linux is not so much better in stability as it is in public imagination. Once a month or two, I have KDE freeze so hard I barely can recover without reset (sometimes can't). Still can't find the reason. Hardware seems to be okay, it is an expensive gaming laptop.
Also, out of RAM can lead to GUI crashing even with swap enabled. True, this is unrelated to servers, which run without GUI, but still weird that it happens.
Also, when my laptop runs for weeks without reboot, it has issues with VRAM being occupied and not freed for some reason, which causes game crashes. With all the talk about Linux infinite uptime I expected better.
However, I almost can't complain about my homelab server. If there is enough disk space, it can run for months. Unfortunately, lately there are so many patches, I need to reboot almost each day (at least podman-hosting VM, if not proxmox host)
For the last half a year I run my homelab, I only had one suspicious shutdown, for which I am not sure I understand the reason well.
Hard freezes are generally related to hardware. Specifically it usually means interrupt processing has stopped, which usually means a driver issue.
Alot of 365 is using Exchange/Sharepoint/Skype for Business code bases which were Windows specific. With Exchange, there was specifically role called "Client Access" which clients connect, the system looks up which backend server has your mailbox and proxies your connection there. Reading between the lines, my guess is this system was reused for Sharepoint/Teams/Office and they could swing off of it to secondaries but Exchange is hard locked on it.
Microsoft does greenfield new stuff on Linux mostly.
Yeah it's pretty crazy that Microsoft, a company most well known for it's Windows operating system, would use that same operating system on its servers.
It's is crazy. Given the fact they are running Linux on their cloud. https://azure.microsoft.com/en-us/products/azure-linux/
That is not a Linux server running MS services. That is a Linux server developed by Microsoft for customers who want a Linux server for their own service infrastructure
Those guys are psycho-fanatics of dogfooding their own services. In the last decade or so, they have hesitantly entertained Apple ecosystems.
The previous know good state?
Windows 10?
7?
XP?
95?
They just postponed an improvement temporarily to reevaluate it later, in fact.
Here, I converted it to corporate speak for you.
This MS rando on X just leaking his 'abridged' version instead of letting the official account post the report/blog later or whatever? Gotta be 'first' right bro, and now the source of every rabid tech blog for the rest of the day. bah