Lots of apologia for Github here. Aside from the fact that defending a billion-dollar company is a bit strange; especially one that is steward to the the overwhelming majority of open-source software.
Maybe that's good-will doing the work? For me it's always been a sour pill to swallow that I have to buy in to a large companies internal politics and practices in order to work on projects I love. I don't feel like I owe them anything.
Especially if they can't hold up their end of the deal.
Unfettered access to the world's software repositories, for the princely sum of a bucketload of Azure credits.
Let me ask the question in reverse: what do you have against them such that the fellow human beings struggling to maintain their operations don’t deserve even a modicum of kindness, respect, and good will? Are you unable to separate the business from the hard working people behind it?
It’s not like they don’t know that people like us are counting on them: they recognize that their service is the “dial tone” for much of the world’s software development capability. They are keenly aware of the impact.
What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?
If I to hire a contractor to redo my roof, and that roof leaks, whether they worked hard or not is immaterial. They did not do the task in they were paid to do. I'm not going to buy their services again just because their shingles guy was particularly charming.
MS has talented engineers, but that's a complete misdirection. Github is a service in decline: there is nothing wrong with criticizing them.
A corporation is not a person. If your organization cannot handle the load, then you need to adjust your practices. The organization needs to prioritize their paying users. The organization needs to shift people from new features to keeping the lights on. And maybe the organization needs to find another strategy to manage its azure transition.
#hugops is to your coworkers, not to the nameless big-corps who can't maintain a service for paying customers. You should be raising a shitstorm when things you pay for aren't reliable or unusable.
Hot take, if it's traffic is causing issues, throttle your free-tier, pause signups, or stop giving out free things (like runner time).
>What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?
Would you feel the same way about a colleague who kept causing downtime in your product again and again, seemingly without making any progress in addressing whatever issue was causing their repeated mistakes?
There are web applications out there that are far more complex than GitHub but have much less downtime. It's not like they're facing an unsolvable problem.
You don’t know that it was “their mistake.” Unless you’ve personally successfully scaled a suite of nontrivial services equivalent to GitHub’s to accommodate an unexpected 14x increase in traffic, you respectfully have no basis for such an assertion.
You could argue the scales are different, but computers are also faster now.
So, argument to credentialism out of the way... What should we do as consumers if a provider that is a defacto monopoly due to network effects stops functioning?
Yeah, they should be testing for that, right? I think there's a lot of people reading comments like yours and thinking, is this person a paid shill or what?
The earn bucket loads of money, they should be planning for exactly that. And testing for it via load testing every day.
Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders.
There are two options, either they are lousy at their jobs, or they are incapable of pushing back against unrealistic demands. Neither is a good indicator of their skill and talent as engineers.
I know I am speaking from a position of some privilege, but I have previously left workplaces that did not allow me to practice good engineering, and I do expect others to do so.
In a SWE job market like this, do you really want to be seen as the "conscientious objector"?
There are literally thousands of people who are ready to ride up the totem pole, it would not be a difficult decision for a bad manager to swing his axe and replace the new head
I think it depends if you pay them money. If you do, then you should indeed have strong expectations towards them and hold them accountable. If they provide a free service to you, then it's still reasonable to feel upset, but at the same time you get what you pay for.
I'm surprised at how little the perception of GitHub changed post-acquisition. Coupled with WSL, it almost balanced things for a lot of people and put Microsoft back in the "benefit of the doubt" column. This is undoing a lot of that, on top of the operational costs. Suddenly the bad press is more noticeable and harder to ignore.
I think its the fact that people have used the software for so long that they feel emotional to it (Hashimoto crying tears of sadness when he decided to move ghostty away from github) and there is completely nothing wrong about it as we are emotional human beings.
But, you are right in the sense that, Github has failed to accept its part of the deal which is actually to just be a usable place. People HAVE previously tolerated so much AI slop and slowness in github's UI just because of its reliability but this downtime is like the Github's achilles heel.
At some point, I recommend people to accept this and move to more healthier alternatives, there is also an momentum. For example, the only reason I joined github was that I wanted to join codeberg but so many of projects used github and involved sign in with github that I finally gave in into github and I had thought that codeberg is so good but nobody is gonna come here because of the network effects but the tide is turning and I hope more people look into codeberg and healthier alternatives.
> Aside from the fact that defending a billion-dollar company is a bit strange
More than a bit strange. This is an HNism that I'll never get. Why would you go to the comment section anywhere to passionately try to defend the honor of a trillion dollar company, unless 1. you're being paid to astroturf or 2. you own that company's stock? Satya Nadella isn't going to read a post here and say, "Gosh, how nice of that commenter! I'm going to send him some Microsoft stock as a show of appreciation for him defending us online!" I don't think I'll ever understand company-fanboys.
This is a real business continuity issue for us. We’re kinda stuck with GitHub Enterprise but we may need to move from cloud to on-premises if this keeps up.
On the other hand when you have a reasonably complex deployment it's easy to get swamped with dashboards showing CPU, Memory, I/O, application-metrics, signups, active users/sessions, etc.
Instead it's nice to think about how you can express the state of a complete system as a single number. It might be you divide active user sessions by database-connections, and then scale by memory capacity.
But as a single digit you can then get used to normal ranges, and have it always visible somewhere obvious. A single number won't show details, but when it changes you can go look at the specific metrics. It's a cute shorthand, and it can work well as a basic "are we normal" check.
It’s obviously a meme website, the meme is more funny when the number isn’t high. Anyone looking for actual accurate info would go to the real status page.
splitting the status page like they do, to the point where it is only a bit of humourous exaggeration to say that they track broken `git push` and `git pull` separately, is a sleight of hand / accounting / SLA-fudging that we should not excuse
there is a subset of the site that pretty much everyone uses — git, issues, pull requests, actions — and if any part of that is broken then the site is broken and the status page should indicate how often this happens
> splitting the status page like they do, to the point where it is only a bit of humourous exaggeration to say that they track broken `git push` and `git pull` separately, is a sleight of hand / accounting / SLA-fudging that we should not excuse
This is a pretty ungenerous take. You could look at it the other way: if I don't use actions then it's useful for me to know that only actions are broken, and I can continue in my normal usage. If you bundle everything up then the status page is reporting an unhelpful false positive for me.
I think the correct middle ground is a site that lets you select the parts of the platform you rely on and ignore the others. For example, GitHub is "down" for me when I can't push, process PRs, or release packages, but I don't care about Actions or AI features.
You’re kind of an outlier - nobody wants AI but Actions are core for tons of workflows and deployment pipelines. Everyone bought into the “only robots can deploy” mantra (correctly IMO, it’s a huge time and friction saver) only to be bit in the ass by the platform being so u reliable they can be stuck for days without deploys.
14x is insane, especially since the quality and quantity of IRL software has barely budged.
One could hope that we'd use these newfound agentic coding powers to actually realize value, improve quality, etc. Instead I see enshittification and stagnation. What are we even doing with all these tokens?
If it can't provide the service, it should stop selling until it can.
This is like the AOL dialup busy signal fiasco of the mid-90's all over again. Except this time, instead of getting mad, people are making excuses for the poor, beleaguered trillion-dollar company.
I really don't understand people saying that this is due to AI commits and it is all the volume's fault.
A volume increase that is a single order of magnitude (which 14x is) should not result in this level of failures.
When I compare what Github does and the volumes vs social media companies, payment companies, video platforms, etc, it just doesn't make sense that it is just a volume problem.
It looks a lot more like a platform that already has baseline issues that are compounded by increased volume.
With Github going up and down and Ubuntu going up and (mostly) down, there's a lot of time for intra-office sword fighting or whatever, lately. If somebody takes down Claude, everybody's going to have to just go home for the day. (https://xkcd.com/303/)
It's not great. Just talked to a hubber last week. They said everyone inside feels pretty dejected right now, and these posts don't help.
I feel for them -- with AI coders submitting 25 PRs within an hour of an issue being filed, GitHub bears the brunt of that along with the maintainers. That's a lot of work that gets done with each PR.
But the amount of compute needed to serve is not very high. It's all text. The amount of bandwidth and compute needed to serve a Netflix or YouTube is far far harder and they managed just fine.
I don't believe they pay top of market, but even if they did, it's possible to make a lot of money and still feel bad when you have a sense of ownership and responsibility to the users of your service.
That being said, 300k TC for E4 is still pretty good. Plus the RSUs have gone up like 60% in the last several years so that 300k package from a few years ago is maybe 350k or more by now.
My point is that they are compensated well. They should be feeling pressure to get this stuff right when their product is core infrastructure for a majority of the digital products that exist today.
I just dont really buy the explanation though. It seems so solvable to hack a throttle or something in place, especially for non-paid plans. The cracks were also showing before AI hit the scene.
Im not saying this is the end-game solution but absolutely they could have put temporary safeguards in place while they "figure it out" if it _really_ is just AI driven slop setting their computers on fire.
The whole "anyone can submit a PR" thing has been a UX issue from day one. That probably needs to go away, and I doubt anyone would really miss it. Where Github could help is by providing a means to build trust that doesn't involve random unknown people slinging code at projects.
That's a hard problem! I don't know. But when we select colleagues we build trust before we let them in the building by interviewing them, looking at their work, checking their references. So maybe there's some sort of analogous process that isn't just "here's a big PR, look at it" would be useful? If there was such a process, maybe that kid could go through it and become trusted.
Lots of apologia for Github here. Aside from the fact that defending a billion-dollar company is a bit strange; especially one that is steward to the the overwhelming majority of open-source software.
Maybe that's good-will doing the work? For me it's always been a sour pill to swallow that I have to buy in to a large companies internal politics and practices in order to work on projects I love. I don't feel like I owe them anything.
Especially if they can't hold up their end of the deal.
Unfettered access to the world's software repositories, for the princely sum of a bucketload of Azure credits.
Let me ask the question in reverse: what do you have against them such that the fellow human beings struggling to maintain their operations don’t deserve even a modicum of kindness, respect, and good will? Are you unable to separate the business from the hard working people behind it?
It’s not like they don’t know that people like us are counting on them: they recognize that their service is the “dial tone” for much of the world’s software development capability. They are keenly aware of the impact.
What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?
When did OP blame the people involved personally?
If I to hire a contractor to redo my roof, and that roof leaks, whether they worked hard or not is immaterial. They did not do the task in they were paid to do. I'm not going to buy their services again just because their shingles guy was particularly charming.
MS has talented engineers, but that's a complete misdirection. Github is a service in decline: there is nothing wrong with criticizing them.
I have all the empathy for people in the world.
A corporation is not a person. If your organization cannot handle the load, then you need to adjust your practices. The organization needs to prioritize their paying users. The organization needs to shift people from new features to keeping the lights on. And maybe the organization needs to find another strategy to manage its azure transition.
OP didn’t blame the staff. His focus is on the company.
Invoking individual workers well-being to defend a billion dollar company is also very strange.
#hugops is to your coworkers, not to the nameless big-corps who can't maintain a service for paying customers. You should be raising a shitstorm when things you pay for aren't reliable or unusable.
Hot take, if it's traffic is causing issues, throttle your free-tier, pause signups, or stop giving out free things (like runner time).
>What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?
Would you feel the same way about a colleague who kept causing downtime in your product again and again, seemingly without making any progress in addressing whatever issue was causing their repeated mistakes?
There are web applications out there that are far more complex than GitHub but have much less downtime. It's not like they're facing an unsolvable problem.
You don’t know that it was “their mistake.” Unless you’ve personally successfully scaled a suite of nontrivial services equivalent to GitHub’s to accommodate an unexpected 14x increase in traffic, you respectfully have no basis for such an assertion.
I have.
You could argue the scales are different, but computers are also faster now.
So, argument to credentialism out of the way... What should we do as consumers if a provider that is a defacto monopoly due to network effects stops functioning?
https://news.ycombinator.com/item?id=47947719
https://www.linkedin.com/in/jharasym/
Yeah, they should be testing for that, right? I think there's a lot of people reading comments like yours and thinking, is this person a paid shill or what?
The earn bucket loads of money, they should be planning for exactly that. And testing for it via load testing every day.
Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders.
> Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders
Genuinely could use a refresher here.
Executives have made a choice to not pay for top talent at Microsoft Azure and Github.
Would you consider telling this to the people working at GitHub directly? I’m sure they’d appreciate your evaluation of their skills and talent.
There are two options, either they are lousy at their jobs, or they are incapable of pushing back against unrealistic demands. Neither is a good indicator of their skill and talent as engineers.
I know I am speaking from a position of some privilege, but I have previously left workplaces that did not allow me to practice good engineering, and I do expect others to do so.
In a SWE job market like this, do you really want to be seen as the "conscientious objector"?
There are literally thousands of people who are ready to ride up the totem pole, it would not be a difficult decision for a bad manager to swing his axe and replace the new head
yes I would tell them "you're underpaid, if you can, come to a company that appreciates your talents more".
What you said came across as an adverse judgment of their skill and talent. Is that not what you meant?
I think it depends if you pay them money. If you do, then you should indeed have strong expectations towards them and hold them accountable. If they provide a free service to you, then it's still reasonable to feel upset, but at the same time you get what you pay for.
Defending a multi-trillion dollar company you mean (Microsoft).
I'm surprised at how little the perception of GitHub changed post-acquisition. Coupled with WSL, it almost balanced things for a lot of people and put Microsoft back in the "benefit of the doubt" column. This is undoing a lot of that, on top of the operational costs. Suddenly the bad press is more noticeable and harder to ignore.
As far as I'm concerned, any benefit of the doubt I might have had for Microsoft is gone after this debaucle: https://news.ycombinator.com/item?id=47989883
there are two groups of people willing to die defending [billion-dollar company]: HN users and Nintendo fans
Apple, clothing brands, even some Microsoft.
Using "apologia" here is pretty embarrassing.
> Maybe that's good-will doing the work?
Of course. GitHub has been an enormous gift to the open source community. Arguably more than Git itself. They deserve a lot of good will.
they are not the non-profit. they make money of it and devs expect certain kind of service in return. GH failed to deliver on the service expectation.
https://xkcd.com/1150/
Oversimplification.
You're right, but that GitHub is dead.
Also, the former stewards of that open source goodness sold it to Microsoft for a cheap buck.
Any goodwill they earned has been spent.
I think its the fact that people have used the software for so long that they feel emotional to it (Hashimoto crying tears of sadness when he decided to move ghostty away from github) and there is completely nothing wrong about it as we are emotional human beings.
But, you are right in the sense that, Github has failed to accept its part of the deal which is actually to just be a usable place. People HAVE previously tolerated so much AI slop and slowness in github's UI just because of its reliability but this downtime is like the Github's achilles heel.
At some point, I recommend people to accept this and move to more healthier alternatives, there is also an momentum. For example, the only reason I joined github was that I wanted to join codeberg but so many of projects used github and involved sign in with github that I finally gave in into github and I had thought that codeberg is so good but nobody is gonna come here because of the network effects but the tide is turning and I hope more people look into codeberg and healthier alternatives.
> Aside from the fact that defending a billion-dollar company is a bit strange
More than a bit strange. This is an HNism that I'll never get. Why would you go to the comment section anywhere to passionately try to defend the honor of a trillion dollar company, unless 1. you're being paid to astroturf or 2. you own that company's stock? Satya Nadella isn't going to read a post here and say, "Gosh, how nice of that commenter! I'm going to send him some Microsoft stock as a show of appreciation for him defending us online!" I don't think I'll ever understand company-fanboys.
This is a real business continuity issue for us. We’re kinda stuck with GitHub Enterprise but we may need to move from cloud to on-premises if this keeps up.
not joking, is there a github repo for this project?
I don't think aggregating the whole platform into one number is fair. It's like adding the whole aws into one number
On the other hand when you have a reasonably complex deployment it's easy to get swamped with dashboards showing CPU, Memory, I/O, application-metrics, signups, active users/sessions, etc.
Instead it's nice to think about how you can express the state of a complete system as a single number. It might be you divide active user sessions by database-connections, and then scale by memory capacity.
But as a single digit you can then get used to normal ranges, and have it always visible somewhere obvious. A single number won't show details, but when it changes you can go look at the specific metrics. It's a cute shorthand, and it can work well as a basic "are we normal" check.
It’s obviously a meme website, the meme is more funny when the number isn’t high. Anyone looking for actual accurate info would go to the real status page.
splitting the status page like they do, to the point where it is only a bit of humourous exaggeration to say that they track broken `git push` and `git pull` separately, is a sleight of hand / accounting / SLA-fudging that we should not excuse
there is a subset of the site that pretty much everyone uses — git, issues, pull requests, actions — and if any part of that is broken then the site is broken and the status page should indicate how often this happens
> splitting the status page like they do, to the point where it is only a bit of humourous exaggeration to say that they track broken `git push` and `git pull` separately, is a sleight of hand / accounting / SLA-fudging that we should not excuse
This is a pretty ungenerous take. You could look at it the other way: if I don't use actions then it's useful for me to know that only actions are broken, and I can continue in my normal usage. If you bundle everything up then the status page is reporting an unhelpful false positive for me.
I think the correct middle ground is a site that lets you select the parts of the platform you rely on and ignore the others. For example, GitHub is "down" for me when I can't push, process PRs, or release packages, but I don't care about Actions or AI features.
You’re kind of an outlier - nobody wants AI but Actions are core for tons of workflows and deployment pipelines. Everyone bought into the “only robots can deploy” mantra (correctly IMO, it’s a huge time and friction saver) only to be bit in the ass by the platform being so u reliable they can be stuck for days without deploys.
Github has far less services and regions that AWS.
quite literally kicking github while it's down!
That purple to blue gradient is the emdash of css.
Supposedly commits on GitHub are up 14x YoY.
They are getting spammed by AI agents?
Yes. There’s no other explanation for 14x, that’s nuts.
Is it spam when they’ve been pushing for this shit and putting AI prompt everywhere fir a year or more?
Commits or pushes? Commits aren't really a worthwhile source of measurement in terms of load.
14x is insane, especially since the quality and quantity of IRL software has barely budged.
One could hope that we'd use these newfound agentic coding powers to actually realize value, improve quality, etc. Instead I see enshittification and stagnation. What are we even doing with all these tokens?
The same thing we’ve done with every other productivity increase in a world based on unfettered growth: garbage.
Supposedly commits on GitHub are up 14x YoY.
So?
If Microsoft can't scale, who can?
If it can't provide the service, it should stop selling until it can.
This is like the AOL dialup busy signal fiasco of the mid-90's all over again. Except this time, instead of getting mad, people are making excuses for the poor, beleaguered trillion-dollar company.
>If it can't provide the service, it should stop selling until it can.
You literally cannot buy GitHub Copilot right now [1].
1: https://github.com/features/copilot/plans
Precisely.
If Microsoft can't scale something like Git 14x, then the problem is with Microsoft.
I really don't understand people saying that this is due to AI commits and it is all the volume's fault.
A volume increase that is a single order of magnitude (which 14x is) should not result in this level of failures.
When I compare what Github does and the volumes vs social media companies, payment companies, video platforms, etc, it just doesn't make sense that it is just a volume problem.
It looks a lot more like a platform that already has baseline issues that are compounded by increased volume.
Meanwhile, my local Gitlab install just hums along no issues.
With Github going up and down and Ubuntu going up and (mostly) down, there's a lot of time for intra-office sword fighting or whatever, lately. If somebody takes down Claude, everybody's going to have to just go home for the day. (https://xkcd.com/303/)
Microsoft is causing Github incidents when Azure data-centers are too hot and they need to make room for Palantir's workload.
This wouldn't surprise me at all, but I'd appreciate evidence to that effect.
Love. Hope Github is a relic of the past inside 12 months
Should be 0 today AFAIK
EDIT: I’m a moron, lol.
it already is. re-read?
Yes, it is, my bad. I was on my way to delete my comment actually! Oh well, too late now… (:
I wonder what morale is like at github. This is like gamer level hating
It's not great. Just talked to a hubber last week. They said everyone inside feels pretty dejected right now, and these posts don't help.
I feel for them -- with AI coders submitting 25 PRs within an hour of an issue being filed, GitHub bears the brunt of that along with the maintainers. That's a lot of work that gets done with each PR.
But they need to make some changes quickly.
But the amount of compute needed to serve is not very high. It's all text. The amount of bandwidth and compute needed to serve a Netflix or YouTube is far far harder and they managed just fine.
they also aren't using azure. IDK what youtube is on, but netflix has actually faced their problems and found solutions (freebsd, mostly)
They should migrate to AWS.
Its webscale
"AI coders submitting 25 PRs within an hour of an issue being filed, GitHub bears the brunt of that....".
What "brunt"? These are not large numbers.
I wouldn't feel too bad for them with their top-of-market comp and valuable RSU packages.
I don't believe they pay top of market, but even if they did, it's possible to make a lot of money and still feel bad when you have a sense of ownership and responsibility to the users of your service.
You missed my point.
GitHub doesn't pay top of market.
You're right.
That being said, 300k TC for E4 is still pretty good. Plus the RSUs have gone up like 60% in the last several years so that 300k package from a few years ago is maybe 350k or more by now.
My point is that they are compensated well. They should be feeling pressure to get this stuff right when their product is core infrastructure for a majority of the digital products that exist today.
I just dont really buy the explanation though. It seems so solvable to hack a throttle or something in place, especially for non-paid plans. The cracks were also showing before AI hit the scene.
Im not saying this is the end-game solution but absolutely they could have put temporary safeguards in place while they "figure it out" if it _really_ is just AI driven slop setting their computers on fire.
The whole "anyone can submit a PR" thing has been a UX issue from day one. That probably needs to go away, and I doubt anyone would really miss it. Where Github could help is by providing a means to build trust that doesn't involve random unknown people slinging code at projects.
Any sort of trust requirement would break the entire model and cause some serious inequality.
How would a random kid in a 3rd world country ever get noticed enough to enter a trust circle, for example?
That's a hard problem! I don't know. But when we select colleagues we build trust before we let them in the building by interviewing them, looking at their work, checking their references. So maybe there's some sort of analogous process that isn't just "here's a big PR, look at it" would be useful? If there was such a process, maybe that kid could go through it and become trusted.
Becoming a joke is the one think that could end the GitHub monopoly.
GitHub is not a monopoly. It never has been. You've always been able to self-host or you can use gogs, gitea, gitlab, bitbucket, you get the idea
Holy bootlickers Batman.