Github are you joking?

GitHub just had its magnum opus of a bug. Now, I've heard lore that people have dropped databases, just some showstopper bugs, but I did not think it was possible for GitHub to cease being Git. Now, we're going to go through what actually happened, but more so I just have a lot of questions as to how this was even possible. All right, so before we begin about GitHub ceasing to be Git, let's first go over a bit of lore because it's important. In 2005, the creator of Linux decided that he was going to take the Brendan Ike challenge, which is to create a programming language in 7 days, but instead create version control in 10. And thus, Git was born. Upon seeing this beautiful distributed version control, software developers did the only thing that made sense. Immediately create a single point of failure. GitHub became so popular that other services decided to use the hub naming convention. Of course, the most popular being DockerHub where you can store all your your Dockers at Microsoft seeing that GitHub attracted developers developers developers decided to purchase GitHub for $7.5 billion and proceeded to generationally fumble the bank. Microsoft, you could have done like nothing. Like if you would have just left GitHub, it would have operated better than whatever has happened today. But the impressive part is that Microsoft has discovered the fourth E in the triple E strategy of course which is embrace, extend, extinguish and shitify. Yesterday we had a regression in the merge Q behavior where in some cases squash or rebase commits were generated from the wrong base state making earlier changes appear reverted in branch history. you had one job, just one singular job, GitHub, which is of course to be Git. Now, that's only about half the story because it really isn't kind of actually illustrating what happened. Now, here's a better story. On April 23rd, GitHub's merge Q started silently reverting code on customers main branches. Now, reverting is actually a Git operation. It did not revert code. That's very important to understand. Not a handful of lines, in some cases thousands. Nothing looked wrong. A PR with a plus 29 minus 34 diff got reviewed, approved, and cued. What actually landed on main was a single commit with plus 245 minus 1,137. Thousands of lines of unrelated already shipped code were quietly removed. Every merge that followed went in on top of that broken history. That means that you would look at the PR. your PR would actually look correct. And when you hit the merge button, what actually went in was not what you hit merge on. This is absolutely diabolical behavior. I mean, to be lied to, like the UI just telling you no, everything's good, man. This is exactly what's going to happen. And for you to have no idea is crazy. I still have a lot of questions, but this is roughly how it's being reported. Now, we do not have the official GitHub kind of postmortem that explains exactly what went wrong. This is just what is being purported by large companies. Effectively, what happens is you have history that looks like this in git. And this would be your main branch. So, let's just say you have four commits on your main branch. And for whatever reason, you branched off back here to make your new branch and then you've added two commits. So the actual root cause of what would happen is that when you'd go to merge, what would end up happening is they would run something called a merge Q, which would take your two merges, which if you just drew this in a straight line would be three circles because you'd have where you branched from and your two changes, and it would squash these two into a singular change. So they are just one single branch. This is actually a whole new branch that's uh created internally in one of Microsoft's CI. Let's just call this singular squashed merge A. What should happen is that A should be merged to the tip of Maine. But what actually ended up happening is that they would take this history right here and overwrite whatever was on main with this right here. So that means if this thing was called B, that means what would be on main now would simply be B to A and that is it. That means these three commits were just gone. Now this obviously caused teams to scramble all afternoon. There are hundreds of tweets of people saying, "Hey, I had to spend all afternoon or multiple days trying to figure out how to detangle whatever has happened and what is actually missing from all of their PRs." GitHub, of course, has said there's only 2,84 poll requests that were messed up out of 4 million merged. Then later on said there was only 800K PRs. Either way, that's the difference between 007% and 3 plus%. It's a big difference. So, which one is it, GitHub? And obviously even more confusing thing if you go to April 23rd. That's weird. I don't Do you see Do you see like some major outage or a partial outage going up there? No, they don't show anything went wrong on the 23rd. Obviously, the reason for that is that the only downtime was you and your team trying to figure out how to undo what just happened. But I also just have a lot of questions and maybe you guys can help me on this one. So the first thing I have is that if this is B and this is A and this is the temporary branch that we have created. If history looked like this, wouldn't pulling down cause you to have some sort of hey the remote branch is like out of sync with your local branch, meaning your two histories have diverged? You wouldn't even be able to pull down main anymore. Or the other option is that instead of just simply putting a right here, it actually had uh revert commits, meaning that it would have these three commits right here, and then it would have another commit that was like, hey, I'm reverting these three commits right here. And then you would have something like a, which would make this not too difficult to be able to go, okay, well, here's the only revert we have. Unreverted, which is not like how anything sounds like. But even more importantly, git is still git. You cannot change history like this easily. Does that mean internally is Microsoft like pushing with force? Right? Because the only way that you could push this right here, this history onto main would be that you have to force push because you're changing history. And if that's the case, if that's the case that they're force pushing, makes me kind of nervous. Like what other issues are there? What other things have been silently happening? And of course, this there's no shot it was 2,84 total. Please double check this stat. We probably had 200 plus as a single customer. So, I'm actually just pretty curious. Did you lose any commits? Are you using the kind of the auto squash feature and thus you have things gone? Also, this has opened up the most legendary excuse of all time for the next year. When any company gets hacked and stuff gets leaked or something goes horribly wrong, they can go, "Oh, yeah. Well, actually, uh, we actually did implement perfect security. We actually had this exact situation already covered, but GitHub decided to silently drop our commit, brother. It's not our fault, it's GitHub's. It's also funny because there's probably a bunch of vibe coders that they just simply just kept on going on with their day. Some weird thing happened and they're getting they're just like, "Oh, fix it. Oh, AI, just fix it." And then when a bug appeared, they're like, "Stupid AI always reverting previous changes. Hey, fix that one. Now again, you know that happened. Absolutely beautiful. And the worst part is for once it actually wasn't AI. They're just they're just going to be blamed for part of this for a whole group of people. Actually, I mean, to be fair, it could have been AI. It could Hey, it could have been. There's some there's some reports going around that it was a faulty feature flag. Of course, Hacker News had its heyday and people were viciously discussing what happened to GitHub. Four people spend hours putting our repo back together at my company after this. GitHub has been unreliable and now they are breaking core tenant of what I expect from this service. Cortenant, you mean tenant, not tenant. You must be fun at parties. I don't care to associate with people who are offended by being corrected. They can saw off and I never go to what you Americans call parties. Whenever you use quotes and they are directly followed by punctuation, you must include the punctuation within the quotes. Parties period reply. Don't you love hacker news? It's just such a lovely place where you see really thoughtful and amazing discussions about topics. But at the end of the day, we still do not have the exact uh postmortem. But we don't know the exact uh cause or reasoning or will GitHub even provide us with the exact details. I don't know. So if anybody can answer these questions, does Microsoft actually use a force underneath the hood to be able to align histories? Were there revert commits, which I don't think there was because nobody is saying that on the internet, wouldn't this cause all sorts of history being absolutely destroyed? Wouldn't this be kind of easy to be able to kind of resolve? Maybe, yeah, it would take a little bit of time. And maybe you have to ask Claude Opus for, you know, Mythos to be able to resolve it. But you should be able to get this done pretty quick. And really, was it only 2,800 PRs? Is that is that it? I mean, Mr. 200 over there doesn't seem to make it that that to be the case. the name is that I think the most impressive thing out of all of this is that GitHub created such a monumentously good name that even after GitHub ceased to be Git for a little bit, everybody's probably still going to keep on using it. I mean, now that my friends is one of the most impressive products of all time at this point, it seems like GitHub is more addictive than cigarettes. A gen. Hey, is that HTTP? Get that out of here. That's not how we order coffee. We order coffee via ssh terminal.shop. Yeah, you want a real experience. You want real coffee. You want awesome subscriptions so you never have to remember again? Oh, you want exclusive blends with exclusive coffee and exclusive content? Then check out cron. You don't know what SSH is? Well, maybe the coffee is not for you. in hand. Living the dream.