Rendered at 08:29:40 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
udave 2 days ago [-]
Hope it survive the next few years of codification of all of humanities unfinished cheap ideas.
robin_reala 2 days ago [-]
Why? Letting it burn to the ground will bring about federated git forges faster.
sdcfgy 2 days ago [-]
This is horribly accurate.
zombot 20 hours ago [-]
As if Microslop didn't have enough unfinished cheap ideas already.
Brian_K_White 2 days ago [-]
I love that phrase.
datsci_est_2015 2 days ago [-]
Still getting Internal Server Error despite githubstatus claiming “All Systems Operational” and the incident as “Resolved”.
Premature declaration of Resolution is pretty bad user experience, if I’m being honest…
reddozen 2 days ago [-]
Who needs to wait for the 5xxs return to normal when you can just press end incident and pad your stats!
jerjerjer 2 days ago [-]
What's to pad there? Their one nine?
zombot 20 hours ago [-]
It can only go up from there.
solarpunk_enthu 2 days ago [-]
Could be an honest mistake.
datsci_est_2015 2 days ago [-]
The original incident was probably an honest mistake as well.
herrherrmann 2 days ago [-]
Possible, but highly unlikely in GitHub’s case.
madeofpalk 2 days ago [-]
At this point, a mistake like this is indistinguishable from malice.
2 days ago [-]
benjy3379 2 days ago [-]
[flagged]
freakynit 2 days ago [-]
Spent 5 minutes debugging why my builds were failing... then recalled github is not google when it comes to availability... checked status.github.com ... and voila.. there it is.
antonvs 2 days ago [-]
We had a similar experience recently with Google: a few team members were trying to diagnose an issue with our systems on Sep 1, for quite a bit longer than 5 minutes. Turns out it was a major Gcloud outage for about 4 hours: https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8...
Google may be more reliable overall, but Gcloud at least tends to be more critical to a business that uses it, so it kind of balances out. I'd much rather have a Github outage than a Gcloud one.
embedding-shape 2 days ago [-]
I'm surprised people still rely so much on GitHub. Anything I want to (have to*) be able to deploy on a moments notice, for hotfixes and what not, was moved away from Github like a year ago, because of the horrible uptime. Did others not get the memo yet?
It's not getting better, if anything it's getting worse. Best time to get off GitHub was yesterday, the second-best day to get off is today. Tangled, Codeberg or self-hosted Forgejo (my approach) or Gitea are all good alternative solutions here.
solatic 2 days ago [-]
It's crazy but there's really still so much out there where hosting on GitHub is practically a requirement, with no other solutions on the market. Off the top of my head, stuff I encountered in the last month:
3. Only GitHub allows you to use free minutes on managed Windows and macOS runners, which are essential for building and signing native Windows and macOS/iOS software. GitLab's are in beta and require a paid plan, Buildkite requires you to pay at least $30/user without any bundled macOS minutes and no hosted Windows runners available. And for small shops, keeping the macOS and Windows runners clean is such a stupid time sink, with the small number of minutes involved it's much more preferable to just start with free managed minutes and plan to pay for managed runner minutes later.
steveklabnik 2 days ago [-]
A recent one for me is Anthropic's first party cloud stuff all assumes you're cloning from GitHub.
embedding-shape 2 days ago [-]
In those cases, it's more often than not enough to just have a read-only mirror on GitHub, that gets automatically replicated when your true upstream gets updated. Might mean a bit more machinery, but I'll take that over letting some flaky Microsoft service stop me from delivering what I promise my users.
viccis 2 days ago [-]
It's free and you can get it up and running pretty much instantaneously with a coding agent. That's it.
They really should be partitioning their infrastructure so that their paying customers are not in the blast radius of outages from the legions of repos running their 700 test vibecoded regression suite every time they change a config file.
John23832 2 days ago [-]
I’m sure they’re working on that, but that takes time. It definitely doesn’t happen in the relative “overnight” that this new influx of code spam spun up.
antonvs 1 days ago [-]
> Did others not get the memo yet?
Perhaps we're not so dysfunctional that we regularly need to "deploy on a moment's notice." That's a business smell.
enraged_camel 2 days ago [-]
>> Anything I want to (have to*) be able to deploy on a moments notice, for hotfixes and what not, was moved away from Github like a year ago, because of the horrible uptime. Did others not get the memo yet?
This misses the point in a bad way. You shouldn't have a single deploy funnel to begin with. You should be able to do deploys from multiple places.
embedding-shape 2 days ago [-]
Sure, and backups should be replicated over 3 mediums, and you shouldn't deploy bugs into production, and...
Lots of things we should do, only a few we actually have time to fix. If I were choosing between "Migrating away from GitHub" and "Adding another way to deploy in case GitHub is down", I'd take advantage and do the first, because you'll end up having to replicate SCM, build infrastructure and so on anyway, why not do it properly?
antonvs 2 days ago [-]
> This misses the point in a bad way.
This misses the business reality in a bad way. A typical outage with something like Github is not a big deal for most businesses. Sure, some nerds get annoyed, but that's about it. The extra effort in maintaining multiple deploy approaches for a non-trivial system simply doesn't make business sense in most cases.
jyounker 2 days ago [-]
This is true.
It's also something that's easy to do if you start from the beginning treating your CI/CD system as something that just runs scripts out of your repo.
antonvs 1 days ago [-]
Have you ever worked at a company of any significant size?
openscript 2 days ago [-]
Does 89.99% count as "three nines availability"?
sandeepkd 2 days ago [-]
Of course not, guess you meant that as a satire
> https://www.githubstatus.com/
Their status page records the outage anywhere between 6 to 17 minutes across the different features. The detailed status page clearly has recorded time of when the errors were escalated, when the fix was applied and when it was assumed that the system has recovered. Its clearly a lot more than 17 minutes, infact its more than an hour.
Unless there is something more to it, these status pages feel like a blatant lie, wondering how long before some one actually sues them cause they are publishing/advertising wrong SLAs.
> On September 13, 2026, between 08:43 and 10:44 UTC
The status page records the incident outage duration as 1 hour 12 minutes
freakynit 2 days ago [-]
So, "git push" is down ... how are github engineers gonna push the fix now?
bearjaws 2 days ago [-]
rsync and sftp, like god intended
PackMan97 2 days ago [-]
Might I ask why you aren't working directly on the prod server? God doesn't use staging or dev, he just codes directly on prod.
hdgvhicv 2 days ago [-]
I use dev. It’s a cname to prod.
14u2c 2 days ago [-]
Prodliness is next to Godliness.
bioneuralnet 2 days ago [-]
Are you sure? What if we're living in the staging universe?
overtone1000 2 days ago [-]
The last decade makes me think he needs to modernize his workflow.
2 days ago [-]
Sohcahtoa82 2 days ago [-]
I prefer Dropbox.
redanddead 2 days ago [-]
Like the olden days
ChuckMcM 2 days ago [-]
Nah in the olden days we'd load a tape with patches and the code would read a patch off tape, apply it, then read the next one. lol.
sdcfgy 2 days ago [-]
God uses UUCP
tripleee 2 days ago [-]
in vim over ssh of course.. agents can use vim right?
edoceo 2 days ago [-]
That can, but cannot exit
dodslaser 2 days ago [-]
Yeah, but they keep trying to use Emacs commands...
C-x M-c M-unfuck-git
zrail 2 days ago [-]
Real talk I have a M-x unfuck-this-buffer. All it does is toggle the Unicode input mode that I sometimes accidentally trigger but have never figured out how.
marcosdumay 2 days ago [-]
You'd use tramp... What is way better than the vim's solution.
xydone 2 days ago [-]
git push --force. No time for the lease either
etatester 2 days ago [-]
Easy. GitHub is hosted on Sourceforge.
whalesalad 2 days ago [-]
sudo git push
pkthunder 2 days ago [-]
Make no mistakes.
snihalani 2 days ago [-]
s3/blob storage
Taurenking 2 days ago [-]
[dead]
rapind 2 days ago [-]
Based on the comments in this thread, you'd think that Reddit was down.
iambenm 2 days ago [-]
"Please don't post comments saying that HN is turning into Reddit." [0]
This isn't that imo... do you think it violated the spirit of that rule?
embedding-shape 2 days ago [-]
What do you think parent is trying to say, except that this HN comment section looks like reddit? I'm trying my hardest to apply charitable reading, but even this has limits.
gchamonlive 2 days ago [-]
My understanding as well, as if Reddit was down and redditors took to HN to distill their anger. It's not calling a specific comment like this, it's calling most of comments here like this.
microsoftedging 2 days ago [-]
Perhaps about the quality of the comments. In my opinion it's not really that bad but reddit-style comments are often cheap shots that are slightly off topic, some of which I see present here
jader201 2 days ago [-]
But that's exactly what the HN guidelines are trying to prevent/discourage -- meta discussion about Reddit-quality comments.
It doesn't add to on-topic discussion.
shevy-java 2 days ago [-]
There are many differences IMO. Moderators, for instance, censor a ton on reddit; but the UI is also different. I much preferred old.reddit.com over both the new reddit (which is utter trash) and hackernews (which is simple, but also ... strange and awkward, much more confusing than old.reddit.com - and I hate the fact that after like 5 comments, I get locked out for hours before I can make more comments, that is the worst decision hackernews ever made).
2 days ago [-]
Rapzid 2 days ago [-]
That it looks like reddit is down so a bunch of people used to posting low effort slop over there ended up here posting low effort slop?
This thread isn't "HN" and that comment wasn't saying it was turning into Reddit..
But let's call a spade a spade. I got worried my account would be flagged due to the amount of down voting I was doing.
shevy-java 2 days ago [-]
I don't think length intrinsically denotes quality. One could say that longer sentences show more quality, but I would not even be certain of that either.
The biggest difference I noticed has been between smartphone/tablet users on the one hand, and oldschool desktop computer system users. I belong to the latter group and I think we, as a group, write more, and faster. I'd also like to assume it has a higher quality, but I am not automatically convinced of that either.
Rapzid 2 days ago [-]
Sure, I mean I didn't mention length but to your point:
Reactionary, performative, low effort slop that's out of place.
Unfortunately the latter is on the rise big time and overly prevelant in this thread.
Drupon 2 days ago [-]
Always thought this rule is hilarious because the fact that such comparisons are so overwhelmingly common that they needed to address it with this bitchy little rule (adding one link to an example per word is a classic tell that an internet moderator is spending their highly compensated moderation time Extremely Mad lol).
However, rather than understand the obvious explanation for it, that reddit and HN's karma systems encourage performative comment behavior with people trying to be epic for the peanut gallery, they make the boneheaded decision to just assume it's an "illusion". It's like Dunning-Krueger but for their theory of mind and general emotional intelligence, with the cause being obvious enough that I don't feel I need to name it.
sdcfgy 2 days ago [-]
Nah. I can't see anyone blaming it on Trump, Israel, datacentre water having a memory, their parents, their neighbour, spirits or spez.
gchamonlive 2 days ago [-]
Why bring trump at all into the discussion, if not just to sh*tpost?
sdcfgy 2 days ago [-]
Everything I listed are symptoms not causes...
icantevenhold 2 days ago [-]
Complaining that HN is becoming Reddit is imo even worse than posting Reddit-level comments - those are at least sometimes funny
ofjcihen 2 days ago [-]
“People here have opinions that I don’t like.”
ballsac 2 days ago [-]
[dead]
DaSHacka 2 days ago [-]
I suspect it'll only become more common as Reddit enshittifies further and further. The platform is basically unusable for anyone, thus its suitability as a 'containment zone' for the redditor phenotype goes down with it.
I hope BlueSky makes a Lemmy-style Reddit alternative, just something to satiate people's social media-ified forum fix.
user- 2 days ago [-]
I can attest to the outage starting around 11:05, as that’s when my GitHub auth for Tailscale SSH broke. Of course, the next 10 minutes of troubleshooting and checking GitHub’s status page showed “all systems operational,” so I assumed it was something on my end. Then, it started working again.
And while I was typing this comment it just blipped again. Atleast this time I know why.
I feel dumb for setting up tailscale like 7 years ago using the github auth, time to figure out how to just move to email on that account.
DaSHacka 2 days ago [-]
> I feel dumb for setting up tailscale like 7 years ago using the github auth, time to figure out how to just move to email on that account.
Unless I've missed something, I don't think you can. I also had to use GitHub auth, as it was the least-bad option of the choices given. I wish they just did email/user + pass + TOTP like regular platforms.
Too much work though. Easier to just switch to Nebula or Netbird and self-host the full stack.
stryan 2 days ago [-]
You can't do just an email address, but if you're running your own domain you can do your own OIDC with it. My tailnet auths against my self-hosted KanIDM server.
zrail 2 days ago [-]
There aren't any restrictions anymore on who you can invite to your team. I added a passkey admin account and added other users via email invites. The account owner is still my GitHub account but I pretty much don't use it day to day.
hdgvhicv 2 days ago [-]
1105 utc? I had (self hosted) runners running at 1336 gmt although when I had a quick look it felt a bit odd - jobs ahead finished but hadn’t updated the pr status
user- 2 days ago [-]
11:05 EDT
gwbas1c 2 days ago [-]
What's frustrating is that:
1: The status site says that pull requests are working,
2: A PR I'm working on right now is missing commits,
3: The contact support page errors out.
atsjie 2 days ago [-]
Contact support page is probably just a waste of time even if it did work.
nevir 2 days ago [-]
Definitely not resolved yet.
kingcauchy 2 days ago [-]
Yeah can't push or create PRs yet for me.
knowknow 2 days ago [-]
Embarrassing that a multi-billion dollar corporation that other billion dollar corporations depend on seems to be fine with breaking their clients workflows. Was github always this shoddy or has something changed to cause this many outages?
mjr00 2 days ago [-]
> has something changed to cause this many outages?
AI coding. Which can be interpreted one of two ways:
* The generous way, which is that Github is so overloaded with massive volumes of AI codebases and AI-driven automation that they're hitting a scale they never anticipated; or
* The not-so-generous way, which is that Github itself was one of the first companies to push everyone to AI code as much as possible, which has lead to an eventual breakdown of the stability of the system, as the people responsible for it no longer understand how it actually works, leading to production outages once every few days.
cowlevel 2 days ago [-]
It also coincides with GitHub moving to exclusively Azure infrastructure. Azure is a terrible shit show for reasons that have nothing to do with AI.
Rapzid 2 days ago [-]
That's a correlation but I've never seen a proof of causation. The variable can't be isolated because they also started shipping more features.
Also, the uptick in odd issues post acquisition pales in comparison to the scaling issues they've had over the past 1-2 years. As a matter of scale, trying to link this back to Azure doesn't really square. If anything they'd potentially be in a worse spot without having access to the resources of a massive public cloud..
burningChrome 2 days ago [-]
There were internal warnings about making the move from highly customized MySQL bare-metal clusters to cloud infrastructure because of network latency, moving from physical machines to multi-tenant environments and having to rewrite a lot of their existing customization.
The other issue of trying to quietly move terabytes of information while the site is live hasn't helped either. Add in the challenge of keeping highly customized bare-metal databases in sync with a cloud environment isn't easy.
Its this very very very complex migration that's resulted in a lot of the ongoing issues.
The migration underscores Microsoft’s strategy to unify its AI and developer ecosystems under Azure, bolstering performance and reliability for Copilot and related AI workloads. However, not all GitHub employees are confident in the transition. Internal concerns have surfaced about potential service disruptions, particularly given the complexity of moving GitHub’s massive MySQL clusters, which currently run on custom bare-metal infrastructure.
Outages have become more frequent in recent months, a symptom of GitHub’s growing operational strain. Insiders suggest the platform’s infrastructure - originally designed to handle conventional development workloads - is now being pushed to its limits by large-scale AI integrations and surging user activity.
Rapzid 2 days ago [-]
> There were internal warnings about making the move from highly customized MySQL bare-metal clusters to cloud infrastructure because of network latency
I'm sure there were, every plan has detractors. Azure has bare metal offerings; does GitHub have access to them? Vladimir Fedorov, GitHub CTO, cited constrained capacity in their data centers and accelerating Azure migration as critical for them to deal with the increased AI load.
IDK, but again as a matter of degree the big issues they are having seem more strongly correlated with agentic coding explosion. In fact it motivated their accelerated migration plans.
nosefrog 2 days ago [-]
Haha I hope this is true, it fits my experience on Azure. 0 machines available. If someone from GitHub wants to chime in.
franktankbank 2 days ago [-]
Azure sucks balls. Its support has no clue whats going on.
Give them some slack. Their core product is AI, not version control
embedding-shape 2 days ago [-]
They used to run their own data centers, and while the unicorn (their error page back in the day) was visible sometimes, it wasn't nearly as bad as things are today.
And of course, Microsoft is saying that none of this downtime is at all related to them moving everything to Azure, and also at the same time they'll fix all this downtime by finishing moving everything to Azure.
I wouldn't hold my breath here.
ainar-g 2 days ago [-]
> Was github always this shoddy or has something changed to cause this many outages?
They went woke. Never seen anyone resembling the average dev in their promotion… You can see the correlation in their downtime graph.
balanceiskey15 2 days ago [-]
GitHub added new webhook IP ranges this morning, curious if that's related.
gagan2020 2 days ago [-]
That might be the cause but let's see.
There 4 updates contains webhook mention in 2 of the updates.
"Webhooks is experiencing degraded performance"
benlivengood 2 days ago [-]
Forgejo/Gitea are pretty easy to set up and compatible with GitHub Action syntax. PRs work almost identically. The mildly annoying part is that Kaniko got deprecated by Google and the company that forked and maintains it doesn't publish binaries so you'll have to build it from source yourself if you want isolated non-root builds in your runners. Any of the major LLMs will get it running pretty solidly without much trouble, though.
I combined it with Zot for an OCI registry but there are quite a few choices there.
GH reliability has been a regular question mark lately, and many time issues are not reported publicly too.
In the past 2 months I've seen 3+ instances where GH action were not triggered on commit pushes, leading to silent workflow failures, and seeing GH merge queue being stuck for 3+ hours for production product facing repos
MaoSYJ 2 days ago [-]
We need the opposite at this point, make an article when they had an entire week without incidents
time0ut 2 days ago [-]
Well it is a Wednesday.
I noticed because SSO was down. They don't even report on that as far as I can tell.
cowlevel 2 days ago [-]
gitea, forgejo, gitlab CE, and ssh still working fine
user43928 2 days ago [-]
I never noticed downtime with public GitLab. It's also generally better than GitHub and Actions.
A self-hosted enterprise GitLab instance in Azure, on the other hand, has had outages and performance degradation quite a few times.
mort96 2 days ago [-]
I have to second this, GitLab CI is pretty good. I'll never be on board with infrastructure people's obsession with putting code in yaml but outside of that, the model makes much more sense to me.
A GitLab runner is a machine/VM which can get notified of a GitLab CI job, then run the job's container image (downloading it from GitLab's container registry if necessary), then run the code in the yaml file. You get to control your build environment.
It's so weird that GitHub Actions's model is "have one absolutely gigantic container image which contains everything any build could ever need, and if something's missing, the documented answer is to apt install it every run".
cowlevel 2 days ago [-]
Of course. Azure is the reason GitHub is down so often, too.
hermannj314 2 days ago [-]
Exactly. I'm running forgejo in a container, pushing backups to a private S3 storage, and sent the forgejo team a small liberapay payment.
Maybe 30 minutes to get it all setup and then I moved on with my life.
shimman 2 days ago [-]
What are you using for CI on your forgejo instance? Are there any popular "turn key" solutions out there?
hermannj314 2 days ago [-]
Currently, I am doing firmware development for my clients, so I do not have a CI pipeline. I'm doing build, test, flash manually on the bench.
(about to be hit with rotten vegetables by the purists...)
shimman 2 days ago [-]
ah this is what I do too, I understand there are things like woodpecker but wish there was just some community docker image that handles this all for you.
pixelesque 2 days ago [-]
One of the PRs I eventually managed to open 8 mins ago had a page flash up a generic error page containing a badge error of: "Unexpected token '<', "<!DOCTYPE "... is not valid JSON"
aarestad 2 days ago [-]
99% of the time, this is a server returning a generic 500 HTML response to an HTTP call that was expecting a JSON response.
geooff_ 2 days ago [-]
Actions being down is one thing - I can't even land a PR.
gagan2020 2 days ago [-]
Same here. I wanted to push something on Github and its was giving 500. This is most severe issue in recent times.
vitosartori 2 days ago [-]
PR? I can't even push haha
laurent123456 2 days ago [-]
I can't even create issues (error 500)
zer0x4d 2 days ago [-]
My self hosted Gitlab still at 99.9% uptime since the last time github/gitlab were down.
bob1029 2 days ago [-]
The biggest challenge I have with this is that the outages are still not bad enough to justify the suffering of self-hosting everything or adopting an enterprise account manager. The Pareto front is really annoying here.
My clients would need to experience a proper multi-day outage (e.g. Monday thru Wednesday) to force the real conversation. Executive leadership is generally not interested in risking a migration to an alternative solution unless the current solution is actively and unambiguously engulfed in flames with a non-zero # of casualties involved.
Hosting it yourself is a nightmare if you are trying to cover issues, pulls, project boards, actions, pages, etc. If it were so easy to self-host the GitHub experience, I don't think we'd be in threads talking about how a billion dollar company is fucking it up every week. Self-hosting just the git front end with a web shell is not GitHub. That is a pretty severe case of moving the goal posts. Now, whether or not your business only needs the git SCM piece is a separate conversation. Maybe you shouldn't have been using GitHub in the first place and moving off would be trivial.
The value of GitHub for me descends in roughly this order:
0. The ability for non-developers to interact with the system
1. Issues
2. Pull requests
3. Notifications
4. Enforcement of process over PRs
5. Project management
6. Actions
7. Git SCM
If I didn't care about the first six things I would still be rocking Jenkins and TFS.
ambicapter 2 days ago [-]
Think of it as just enough impetus for everyone to deliver a first-class local development experience.
alexellisuk 2 days ago [-]
I thought it was odd that we'd not had one of these for a few weeks.
A few hours ago GH pages just would not publish at all.. and the status page showed as fine, I guess it's now cascaded into a proper event.
malfist 2 days ago [-]
Not had one for a few weeks? There was an outage on Monday (as usual)
cowlevel 2 days ago [-]
There was one yesterday.
franey 2 days ago [-]
I'm still having trouble pushing to my repo although the issue. is marked as resolved. Maybe the fix needs some time to seep through various servers.
Edit: 4 minutes later and I can push again
odo1242 2 days ago [-]
To be fair to GitHub, outages are pretty short (~40 minutes), and they rarely impact critical parts of the service, nor do they affect one’s ability to work (git offline)
mort96 2 days ago [-]
What... what parts of GitHub would you consider "critical parts of the service" if not git operations, pull requests and CI?
PestoDiRucola 2 days ago [-]
Production has a critical error and we need to push a fix while our whole build pipeline runs on GH CI. Which is not working. Thank God their critical parts never break.
slowin 2 days ago [-]
Not being able to push is pretty critical.
m2ger 2 days ago [-]
git push ending up with Internal Server error is similar to save file ending up with disk full error. yes, it can work after 40 minutes, and one can go to lunch and retry later, but here in Europe the troubles started at the end of a (normal) workday.
godtoldmetodoit 2 days ago [-]
If I can't push my code, I'd say that's a critical issue. And if my Actions won't run (that I pay for), I'd also put that in the critical issue bucket.
I'm going to be investigating self hosting, this is ridicolous.
OrderlyTiamat 2 days ago [-]
40 minutes is an eternity when you can't push to a gitops repo with a PRD issue.
Rapzid 2 days ago [-]
If I had a super critical gitops pipeline that couldn't deal with the odd provider outage I'd be using my cloud providers repo service.
Same as with container registries; prod shouldn't be pulling from Dockerhub.
mschuster91 2 days ago [-]
... and that is (yet another example) why pipeline driven gitops is a really darn bad idea.
lenerdenator 2 days ago [-]
I'm not able to get code pushed, meaning I can't address PR feedback, meaning I can't get things approved, meaning code isn't getting merged to main and thus a ticket isn't getting closed. Sounds pretty critical to me.
mococa 2 days ago [-]
In Mars, right?
angrydev 2 days ago [-]
Good bait
acclay 2 days ago [-]
I'm tired, boss.
bitlad 2 days ago [-]
It started after I removed my primary email in the profile.
Did I break this? Making me insecure about changing anything on github.
wreath 2 days ago [-]
Silver lining is that this slows down the all the crappy half baked vibe coded crap from being shipped.
iercan 2 days ago [-]
first time seeing "git push" on github throwing an "Internal Server Error"
AbstractH24 19 hours ago [-]
It was mostly a hiccup for me
colek42 2 days ago [-]
We kept pushing to GitHub. Pushgate.dev is a gitproxy. It forces your agents to run tests and validates them. We keep the happy path as fast as possible and use Cloudflare's infrastructure for high resilience. Full disclosure: I am the founder of TestifySec; Pushgate is a new product.
tripleee 2 days ago [-]
Three days in a row now right?
aleqs 2 days ago [-]
GitHub is now one of the least reliable services in my stack, I'm in the process of moving everything off and keeping it for public/open projects for discoverability/discussion purposes only.
I've also noticed a meaningful decline in the quality and reliability of several other major software/apps and services - notably Google maps, Spotify, codex. (Is this because of AI?)
Codex, especially, is a strange one - initially I added it beside Claude code thinking it would actually be a higher quality, faster alternative (even if the model/harness is not quite as good at coding as Claude code/opus), and it seemed that way for the first few weeks. At this point though, codex is completely unusable for me - the contexts from different sessions are polluting/mixing with each other, prompt history is mixing/polluting each other. Codex randomly enters safe sandbox mode or whatever multiple times a day per session - it just starts seeing everything as read-only and loses tool and MCP access. It also randomly gets interrupted all the time for no apparent reason - 'your session has been interrupted' - so much for 'autonomous'. And holy shit codex is sooo slow, I'm guessing there are both server side and client side rate-limits, I have the 20x pro plan and I can't even use it up (I use up my Claude code pro 30x in 2-3 days, though to be fair Anthropic usage limits are arbitrary and low), because it is so slow plus all of the issues and constant random stops/blockers that prevent any kind of long-running autonomous work.
Opened a couple codex issues on GitHub for these things - all closed as duplicates. I look at the existing issues - all closed as resolved.
honest question - has anyone moved to cursor origin after outages like this? why / why not?
pullshark91 2 days ago [-]
Was GitHub always like this?
apefulsin 2 days ago [-]
Only since the end of 2025, which is when AI agents got really popular, and also when GitHub moved to Azure. Take your pick of potential causes.
esafak 2 days ago [-]
This is such an opportunity for competitors. I hope they don't squander it.
lenerdenator 2 days ago [-]
The drive to win a market by accumulating customer market share is no longer the main drive behind business ventures in our economy. It has been replaced with the drive to increase equities market valuation. This, at times, contradicts accumulating or retaining customer market share.
Not many people are interested in providing investment funding a meaningful competitor to GitHub. There are sexier, bigger bets out there than "providing a place for version control software to be hosted". The people who are frustrated enough with GitHub to self-host have the ability to do so. The rest of us just get up from our desks, go for a coffee break/walk, and wait it out, because migrating to self-hosted is a PITA and not a core business competency.
They'll keep having outages, and most will keep paying for it. If they're paying for it despite fewer resources being spent on uptime, that means the company's more valuable as an equities market participant. Line go up.
uptownhr 2 days ago [-]
I still cannot push :(
hdgvhicv 2 days ago [-]
Must be a Y in the day
lucas_the_human 2 days ago [-]
Guys...
nharziro 2 days ago [-]
this is getting ridiculous.
x3n0ph3n3 2 days ago [-]
They claim it's resolved, but I'm still seeing issues.
acedTrex 2 days ago [-]
Im so grateful that coding is solved
honeycrispy 2 days ago [-]
Funny, yesterday git operations had an unreported outage too. I couldn't pull changes for ~10 minutes.
sleepybrett 2 days ago [-]
i see they started letting the github engineers ship code again after that shitshow last month.
2 days ago [-]
bossyTeacher 2 days ago [-]
This is becoming way more frequent than a monthly event. Does anyone know why GitHub keeps having incidents?
christophilus 2 days ago [-]
It’s mentioned on most threads, but their traffic and usage is way up due to AI / agentic programming.
Brian_K_White 2 days ago [-]
Which let's not forget that they themselves push, both gh themselves and their parent ms.
Let's also not forget that rate-limiting is a trivial thing and small but non-zero charge is also a trivial thing (and would have the convenient benefit of more than paying for itself).
They actively want the free tiers which make the traffic flood/abuse possible because they want the community/market capture and the data.
None of this is unavoidable burden sat on them from the outside like a solar flare.
They push the use of AI tools including their own, and choose not to do any of the trivial things that any of us in any form of IT knows they could do to rate-limit which wouldn't hurt any normal legit user, even ones that want to use AIs.
They beg people to run wild on their service and then fall over every other day from it, and just keep on begging people to run wild on their service. It's been literally years now. Not some new development.
cowlevel 2 days ago [-]
Because it's vibecoded and because it's running on Azure
rvz 2 days ago [-]
The last time GitHub went down was...yesterday.
GitHub is never going to improve and at this point. You might as well say that the AI chatbots, Tay and Copilot are maintaining the platform and are running it into the ground. No CEO of GitHub to go to as well.
It is time to self host instead of using something as broken as GitHub as I predicted 6 years ago.
mococa 2 days ago [-]
Again?
MadrasTh0rn 2 days ago [-]
Love it
oslem 2 days ago [-]
Color me shocked
nstott 2 days ago [-]
This is so frustrating.
I wish it were easier to migrate away from this tool
coliveira 2 days ago [-]
That's what you get when the bulk of open source is now depending on Microsoft.
Premature declaration of Resolution is pretty bad user experience, if I’m being honest…
Google may be more reliable overall, but Gcloud at least tends to be more critical to a business that uses it, so it kind of balances out. I'd much rather have a Github outage than a Gcloud one.
It's not getting better, if anything it's getting worse. Best time to get off GitHub was yesterday, the second-best day to get off is today. Tangled, Codeberg or self-hosted Forgejo (my approach) or Gitea are all good alternative solutions here.
1. Homebrew really, really wants you to host your taps on GitHub. The docs are very GitHub-focused: https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap as are downstream projects like dist: https://axodotdev.github.io/cargo-dist/book/installers/homeb...
2. Terraform and OpenTofu public registries support only GitHub, see e.g. https://developer.hashicorp.com/terraform/registry/modules/p... and https://opentofu.org/docs/language/modules/develop/publish/#...
3. Only GitHub allows you to use free minutes on managed Windows and macOS runners, which are essential for building and signing native Windows and macOS/iOS software. GitLab's are in beta and require a paid plan, Buildkite requires you to pay at least $30/user without any bundled macOS minutes and no hosted Windows runners available. And for small shops, keeping the macOS and Windows runners clean is such a stupid time sink, with the small number of minutes involved it's much more preferable to just start with free managed minutes and plan to pay for managed runner minutes later.
They really should be partitioning their infrastructure so that their paying customers are not in the blast radius of outages from the legions of repos running their 700 test vibecoded regression suite every time they change a config file.
Perhaps we're not so dysfunctional that we regularly need to "deploy on a moment's notice." That's a business smell.
This misses the point in a bad way. You shouldn't have a single deploy funnel to begin with. You should be able to do deploys from multiple places.
Lots of things we should do, only a few we actually have time to fix. If I were choosing between "Migrating away from GitHub" and "Adding another way to deploy in case GitHub is down", I'd take advantage and do the first, because you'll end up having to replicate SCM, build infrastructure and so on anyway, why not do it properly?
This misses the business reality in a bad way. A typical outage with something like Github is not a big deal for most businesses. Sure, some nerds get annoyed, but that's about it. The extra effort in maintaining multiple deploy approaches for a non-trivial system simply doesn't make business sense in most cases.
It's also something that's easy to do if you start from the beginning treating your CI/CD system as something that just runs scripts out of your repo.
> https://www.githubstatus.com/ Their status page records the outage anywhere between 6 to 17 minutes across the different features. The detailed status page clearly has recorded time of when the errors were escalated, when the fix was applied and when it was assumed that the system has recovered. Its clearly a lot more than 17 minutes, infact its more than an hour.
Unless there is something more to it, these status pages feel like a blatant lie, wondering how long before some one actually sues them cause they are publishing/advertising wrong SLAs.
Another example
> https://www.githubstatus.com/incidents/0rn90wk115q9
> On September 13, 2026, between 08:43 and 10:44 UTC
The status page records the incident outage duration as 1 hour 12 minutes
C-x M-c M-unfuck-git
[0] https://news.ycombinator.com/newsguidelines.html
It doesn't add to on-topic discussion.
This thread isn't "HN" and that comment wasn't saying it was turning into Reddit..
But let's call a spade a spade. I got worried my account would be flagged due to the amount of down voting I was doing.
The biggest difference I noticed has been between smartphone/tablet users on the one hand, and oldschool desktop computer system users. I belong to the latter group and I think we, as a group, write more, and faster. I'd also like to assume it has a higher quality, but I am not automatically convinced of that either.
> Why is GitHub down so often?
Short. Invites conversation. Shows intellectual curiosity.
> It's so over
Reactionary, performative, low effort slop that's out of place.
Unfortunately the latter is on the rise big time and overly prevelant in this thread.
However, rather than understand the obvious explanation for it, that reddit and HN's karma systems encourage performative comment behavior with people trying to be epic for the peanut gallery, they make the boneheaded decision to just assume it's an "illusion". It's like Dunning-Krueger but for their theory of mind and general emotional intelligence, with the cause being obvious enough that I don't feel I need to name it.
I hope BlueSky makes a Lemmy-style Reddit alternative, just something to satiate people's social media-ified forum fix.
And while I was typing this comment it just blipped again. Atleast this time I know why.
I feel dumb for setting up tailscale like 7 years ago using the github auth, time to figure out how to just move to email on that account.
Unless I've missed something, I don't think you can. I also had to use GitHub auth, as it was the least-bad option of the choices given. I wish they just did email/user + pass + TOTP like regular platforms.
https://tailscale.com/blog/passkeys
Maybe running your own OpenID service would work?
https://openid.net/developers/how-connect-works/
Too much work though. Easier to just switch to Nebula or Netbird and self-host the full stack.
1: The status site says that pull requests are working,
2: A PR I'm working on right now is missing commits,
3: The contact support page errors out.
AI coding. Which can be interpreted one of two ways:
* The generous way, which is that Github is so overloaded with massive volumes of AI codebases and AI-driven automation that they're hitting a scale they never anticipated; or
* The not-so-generous way, which is that Github itself was one of the first companies to push everyone to AI code as much as possible, which has lead to an eventual breakdown of the stability of the system, as the people responsible for it no longer understand how it actually works, leading to production outages once every few days.
Also, the uptick in odd issues post acquisition pales in comparison to the scaling issues they've had over the past 1-2 years. As a matter of scale, trying to link this back to Azure doesn't really square. If anything they'd potentially be in a worse spot without having access to the resources of a massive public cloud..
The other issue of trying to quietly move terabytes of information while the site is live hasn't helped either. Add in the challenge of keeping highly customized bare-metal databases in sync with a cloud environment isn't easy.
Its this very very very complex migration that's resulted in a lot of the ongoing issues.
https://hostingjournalist.com/news/microsoft-accelerates-git...
The migration underscores Microsoft’s strategy to unify its AI and developer ecosystems under Azure, bolstering performance and reliability for Copilot and related AI workloads. However, not all GitHub employees are confident in the transition. Internal concerns have surfaced about potential service disruptions, particularly given the complexity of moving GitHub’s massive MySQL clusters, which currently run on custom bare-metal infrastructure.
Outages have become more frequent in recent months, a symptom of GitHub’s growing operational strain. Insiders suggest the platform’s infrastructure - originally designed to handle conventional development workloads - is now being pushed to its limits by large-scale AI integrations and surging user activity.
I'm sure there were, every plan has detractors. Azure has bare metal offerings; does GitHub have access to them? Vladimir Fedorov, GitHub CTO, cited constrained capacity in their data centers and accelerating Azure migration as critical for them to deal with the increased AI load.
IDK, but again as a matter of degree the big issues they are having seem more strongly correlated with agentic coding explosion. In fact it motivated their accelerated migration plans.
And of course, Microsoft is saying that none of this downtime is at all related to them moving everything to Azure, and also at the same time they'll fix all this downtime by finishing moving everything to Azure.
I wouldn't hold my breath here.
The data speaks clearly: https://damrnelson.github.io/github-historical-uptime/
There 4 updates contains webhook mention in 2 of the updates.
"Webhooks is experiencing degraded performance"
I combined it with Zot for an OCI registry but there are quite a few choices there.
In the past 2 months I've seen 3+ instances where GH action were not triggered on commit pushes, leading to silent workflow failures, and seeing GH merge queue being stuck for 3+ hours for production product facing repos
I noticed because SSO was down. They don't even report on that as far as I can tell.
A self-hosted enterprise GitLab instance in Azure, on the other hand, has had outages and performance degradation quite a few times.
A GitLab runner is a machine/VM which can get notified of a GitLab CI job, then run the job's container image (downloading it from GitLab's container registry if necessary), then run the code in the yaml file. You get to control your build environment.
It's so weird that GitHub Actions's model is "have one absolutely gigantic container image which contains everything any build could ever need, and if something's missing, the documented answer is to apt install it every run".
Maybe 30 minutes to get it all setup and then I moved on with my life.
(about to be hit with rotten vegetables by the purists...)
My clients would need to experience a proper multi-day outage (e.g. Monday thru Wednesday) to force the real conversation. Executive leadership is generally not interested in risking a migration to an alternative solution unless the current solution is actively and unambiguously engulfed in flames with a non-zero # of casualties involved.
Hosting it yourself is a nightmare if you are trying to cover issues, pulls, project boards, actions, pages, etc. If it were so easy to self-host the GitHub experience, I don't think we'd be in threads talking about how a billion dollar company is fucking it up every week. Self-hosting just the git front end with a web shell is not GitHub. That is a pretty severe case of moving the goal posts. Now, whether or not your business only needs the git SCM piece is a separate conversation. Maybe you shouldn't have been using GitHub in the first place and moving off would be trivial.
The value of GitHub for me descends in roughly this order:
If I didn't care about the first six things I would still be rocking Jenkins and TFS.A few hours ago GH pages just would not publish at all.. and the status page showed as fine, I guess it's now cascaded into a proper event.
Edit: 4 minutes later and I can push again
I'm going to be investigating self hosting, this is ridicolous.
Same as with container registries; prod shouldn't be pulling from Dockerhub.
Did I break this? Making me insecure about changing anything on github.
I've also noticed a meaningful decline in the quality and reliability of several other major software/apps and services - notably Google maps, Spotify, codex. (Is this because of AI?)
Codex, especially, is a strange one - initially I added it beside Claude code thinking it would actually be a higher quality, faster alternative (even if the model/harness is not quite as good at coding as Claude code/opus), and it seemed that way for the first few weeks. At this point though, codex is completely unusable for me - the contexts from different sessions are polluting/mixing with each other, prompt history is mixing/polluting each other. Codex randomly enters safe sandbox mode or whatever multiple times a day per session - it just starts seeing everything as read-only and loses tool and MCP access. It also randomly gets interrupted all the time for no apparent reason - 'your session has been interrupted' - so much for 'autonomous'. And holy shit codex is sooo slow, I'm guessing there are both server side and client side rate-limits, I have the 20x pro plan and I can't even use it up (I use up my Claude code pro 30x in 2-3 days, though to be fair Anthropic usage limits are arbitrary and low), because it is so slow plus all of the issues and constant random stops/blockers that prevent any kind of long-running autonomous work.
Opened a couple codex issues on GitHub for these things - all closed as duplicates. I look at the existing issues - all closed as resolved.
What a horrendous shitshow.
Not many people are interested in providing investment funding a meaningful competitor to GitHub. There are sexier, bigger bets out there than "providing a place for version control software to be hosted". The people who are frustrated enough with GitHub to self-host have the ability to do so. The rest of us just get up from our desks, go for a coffee break/walk, and wait it out, because migrating to self-hosted is a PITA and not a core business competency.
They'll keep having outages, and most will keep paying for it. If they're paying for it despite fewer resources being spent on uptime, that means the company's more valuable as an equities market participant. Line go up.
Let's also not forget that rate-limiting is a trivial thing and small but non-zero charge is also a trivial thing (and would have the convenient benefit of more than paying for itself).
They actively want the free tiers which make the traffic flood/abuse possible because they want the community/market capture and the data.
None of this is unavoidable burden sat on them from the outside like a solar flare.
They push the use of AI tools including their own, and choose not to do any of the trivial things that any of us in any form of IT knows they could do to rate-limit which wouldn't hurt any normal legit user, even ones that want to use AIs.
They beg people to run wild on their service and then fall over every other day from it, and just keep on begging people to run wild on their service. It's been literally years now. Not some new development.
GitHub is never going to improve and at this point. You might as well say that the AI chatbots, Tay and Copilot are maintaining the platform and are running it into the ground. No CEO of GitHub to go to as well.
It is time to self host instead of using something as broken as GitHub as I predicted 6 years ago.
https://github.com/about