I love how even self hosted workers don't work during these outages - running jobs on their infrastructure being flakey is marginally acceptable, but the API to simply schedule workflows having this availability is mind boggling. Github just doesn't seem like a serious company anymore.
GitHub implements self hosted runners by running a normal runner that passes the environment to your runner and then polls it. That's why they cost as much as the smallest GitHub-hosted runner.
I'm not sure that it's that mind boggling. The entire complexity and value of GHA is not in the runners but the scheduler that the GH frontend hooks into and runners subscribe to jobs from. It's the most likely thing to fall over. '
If the scheduling was self hosted it would be inexcusable but you can always just connect whatever you want to webhooks.
The entire complexity and value of GHA is rent-seeking to keep the lights on for other things.
They have a strong motivation (self preservation) to continue to misunderstand the problem. If they did what is best for us, then we could avoid a substantial fraction of all GH subscriptions by using a FOSS tool to hit the Pareto frontier by replicating just enough GH services to watch commits and PRs.
That "scheduling" is just a git hook and a message queue, maybe with some database updates in between with very clear boundaries that make sharding easy to reason about, assuming they have a sane architecture (they evidently don't)
Something every programmer learns eventually is that it's just silly to guess at how a system works, or should work, without digging into the details. Because you simply don't know what you don't know.
The only statement I'm making about their system is that it appears to be poorly designed, as evident by their poor uptime. The rest is just visible details about what data is involved in performing the task at hand, agnostic to the implementation underlying it.
They never were. They were just really good at maintaining the bluster and backing it up with expensive lawyers. Now that they don't come in claws-first like they did when they were a NASDAQ rocket to the stars, it's easy to see.
Their MO was to court an executive and sell second-rate tools to them before the people who had to use them had a chance to say anything. It doesn't matter how much evidence you can provide to the contrary, once the million dollar deal is signed, you are going to be tasked with finding reasons to say that your executive was shrewd for buying this pile of junk and unfulfilled promises, and not an insane idiot sucking away your job satisfaction as fast as they can.
They did a lot of deals based on how their products would have features their competitors already have 'soon' when they haven't even started them, and a long track record of taking 3 major releases to get from something to good, and then breaking everything again by doing a 4th major release that re-imagined everything and made it horrible again.
I'm not going to claim that Apple was or is a panacea. Apple doesn't use vaporware which is big, and their Cycle of Awful is 2 releases instead of 3. You could afford to skip 1 waiting for the next even-numbered version, instead of being 2 versions behind and getting pressed to upgrade.
Can't wait for the Graphite/Cursor alternative (Origin) to become available. Actions being down for an hour in the middle of the US East workday is crazy. It's super frustrating to be an enterprise customer of Github, seems like we don't get any dedicated resources or additional stability. At this point our only option is self-host github enterprise or move away.
Soon we can start posting "GitHub {x} is up!" instead, that's becoming more news-worthy lol. I know they like to blame AI/scaling, but really they are owned by one of the main hyperscalers, so I find that excuse kind of weak..
Someone needs to work on improving the way Actions deals with resource exhaustion like this. Today it exploded all the CI jobs with "failed" and an inscrutable message about "The job was not acquired by Runner of type hosted even after multiple attempts" then "Internal server error". It should say "Timed out due to no runner available, our apologies, try again later, or pay $$$ here to get priority for runners".
There's a reason I'm browsing HN right now. Dead in the water.
I love how even self hosted workers don't work during these outages - running jobs on their infrastructure being flakey is marginally acceptable, but the API to simply schedule workflows having this availability is mind boggling. Github just doesn't seem like a serious company anymore.
GitHub implements self hosted runners by running a normal runner that passes the environment to your runner and then polls it. That's why they cost as much as the smallest GitHub-hosted runner.
This is no surprise given standard Microsoft operating procedure - https://news.ycombinator.com/item?id=47616242
they walked that back for now at least - self-hosted gh runners are still free: https://docs.github.com/en/billing/concepts/product-billing/...
Well, they're still implemented that way.
Maybe switching to https://github.com/gitlabhq :)
I'm not sure that it's that mind boggling. The entire complexity and value of GHA is not in the runners but the scheduler that the GH frontend hooks into and runners subscribe to jobs from. It's the most likely thing to fall over. '
If the scheduling was self hosted it would be inexcusable but you can always just connect whatever you want to webhooks.
The entire complexity and value of GHA is rent-seeking to keep the lights on for other things.
They have a strong motivation (self preservation) to continue to misunderstand the problem. If they did what is best for us, then we could avoid a substantial fraction of all GH subscriptions by using a FOSS tool to hit the Pareto frontier by replicating just enough GH services to watch commits and PRs.
That "scheduling" is just a git hook and a message queue, maybe with some database updates in between with very clear boundaries that make sharding easy to reason about, assuming they have a sane architecture (they evidently don't)
Something every programmer learns eventually is that it's just silly to guess at how a system works, or should work, without digging into the details. Because you simply don't know what you don't know.
The only statement I'm making about their system is that it appears to be poorly designed, as evident by their poor uptime. The rest is just visible details about what data is involved in performing the task at hand, agnostic to the implementation underlying it.
Microsoft is not a serious company anymore.
They never were. They were just really good at maintaining the bluster and backing it up with expensive lawyers. Now that they don't come in claws-first like they did when they were a NASDAQ rocket to the stars, it's easy to see.
Their MO was to court an executive and sell second-rate tools to them before the people who had to use them had a chance to say anything. It doesn't matter how much evidence you can provide to the contrary, once the million dollar deal is signed, you are going to be tasked with finding reasons to say that your executive was shrewd for buying this pile of junk and unfulfilled promises, and not an insane idiot sucking away your job satisfaction as fast as they can.
They did a lot of deals based on how their products would have features their competitors already have 'soon' when they haven't even started them, and a long track record of taking 3 major releases to get from something to good, and then breaking everything again by doing a 4th major release that re-imagined everything and made it horrible again.
I'm not going to claim that Apple was or is a panacea. Apple doesn't use vaporware which is big, and their Cycle of Awful is 2 releases instead of 3. You could afford to skip 1 waiting for the next even-numbered version, instead of being 2 versions behind and getting pressed to upgrade.
gitlab-ce can run all the integration tests in docker et al.
Sitting on Github these days is the same as sticking to twitter a decade ago, expect next mecha hitler, I suppose.
Can't wait for the Graphite/Cursor alternative (Origin) to become available. Actions being down for an hour in the middle of the US East workday is crazy. It's super frustrating to be an enterprise customer of Github, seems like we don't get any dedicated resources or additional stability. At this point our only option is self-host github enterprise or move away.
WTF. Outage every month if not multiple. Even for paying enterprise plans.
Also, doesn't even have RAG offering.
Incident number 6 for the month of August, notably on August 6th. This pattern does not bode well.
Four of them were regarding AI/Copilot. So that's really just one long running incident and will probably continue to be 50/50 chance of being down.
Soon we can start posting "GitHub {x} is up!" instead, that's becoming more news-worthy lol. I know they like to blame AI/scaling, but really they are owned by one of the main hyperscalers, so I find that excuse kind of weak..
So, Github Actions is currently at two nines of uptime?
are you sure its actually two nines and not zero?
The speed at which GitHub is tanking their reputation is fascinating
How did they get Github Pages into “degraded performance”? Aren’t those just CDN stored static pages?
I believe Pages uses Actions under the hood for publishing, so in my head it sort of makes sense for these things to be affected in the same outage
You are right, didn’t know that.
Seems like the only reliable way to run GHA jobs is to not use their runners. Hope they at least didn’t break self-hosted runners operations
Well hopefully reads aren't affected then...
Still twiddling thumbs over here
Are we even surprised anymore? Since microsoft (the IBM of its time) have taken this over all I can say is ... piss up ... brewery.
Did Microsoft fire all the people working on the GitHub features people actually use?
I mean, Github doesn't have its own CEO anymore, so arguably.
Are there any other CI providers that offer free cloud-hosted Windows runners?
Or has Microsoft made sure (through Windows licensing terms and pricing) that it's not possible to compete with their own CI offering?
Edit: CircleCI seems to offer 750 minutes/month (whereas GitHub offers 1000 minutes/month).
You can always pay for it, use your own machine, use a VM on your own machine, or if you are developing non-slop FOSS, ask Codeberg.
Github Actions are rapidly becoming an argument against dogfooding into your critical path.
I am having their issues, also... Anyway this guys are great!
This has been preventing me from merging a PR for over an hour. I'm really tired of the constant outages GitHub has
Was? Still is
Very frustrating, nothing more I can add at this point.
Ever since Microsoft has taken this over are we even surprised by the downtime anymore? Piss up ... brewery ...
So, it's a normal Thursday?
any 9s left?
there is one 9 of uptime, but the dashboard hasn't been updated yet: https://mrshu.github.io/github-statuses/
Once this goes in, I'd expect to see 89%, which is zero nines. (I'd like to say, "a new low!", but sadly we've had this before)
They're running 9 8s of uptime.
nein nein nein
Someone needs to work on improving the way Actions deals with resource exhaustion like this. Today it exploded all the CI jobs with "failed" and an inscrutable message about "The job was not acquired by Runner of type hosted even after multiple attempts" then "Internal server error". It should say "Timed out due to no runner available, our apologies, try again later, or pay $$$ here to get priority for runners".
Ah. So that's why.
man.. again?
Do we need to say this again? GitHub appears that they are never going to improve. Just look at how GitHub spark has deprecated, now we have this.
After 6 years of this nonsense of "centralizing everything on GitHub", it is not a good idea at all.
You might as well self host like I said before [0].
[0] https://news.ycombinator.com/item?id=22868406