This is bad...

We are investigating unauthorized access to GitHub's internal repositories. While we currently have no evidence of impact to customer information stored outside of GitHub's internal repositories, such as our customers, enterprises, organizations, and repos, we are closely monitoring our infrastructure for followon activity. I I would hope we're all familiar with GitHub at this point, and they've been having a rough time with security and just infrastructure hosting as a whole lately. They don't have a CEO. They don't have particularly good uptime. In fact, the uptime is so bad that this very funny reply popped up. Holy [ __ ] how' the attackers find a large enough uptime window to get in? Yeah, there are a lot of layers to this. from the method in which they were exploited, which spoiler is another thing Microsoft owns, the other exploits happening across GitHub surfaces, in particular npm and the mini shyut attack that's been pwning everybody lately, as well as the core failures that are leading to these types of things happening at Microsoft and at GitHub specifically. And of course, we'll talk a little bit about the potential harm that this would bring to us as GitHub users. As you guys know, I have been having a tough time with GitHub as of late. I am still holding hope that they pull it together, but I'm definitely not betting on it at this point. This has me scared and I'm sure it has them even more so. They call out this below here saying that if any impact is discovered for customers, they will notify them via established incident response and notification channels. This is good to hear because there's a lot of GitHub customers that I care deeply about, one of which is today's sponsor. Let's be realistic. It's not our job to write code anymore. It's kind of our job to review it to make sure all of the things that are changing in a given PR are correct and will ship and work as we expect, but also that they interact with the other projects in our company properly. You probably don't just have one repository unless you're like Google or something. So, how do you know for sure your different projects are working together properly? And how do you keep all of the context that your agents need to know this? Well, if you're not using Code Rabbit, I have no idea how you're doing that because these guys really get it. They have best-in-class AI code review, which is great on a poll request and all, but it goes further than that. I feel like I have had to spend so much time just reading the PR, noticing some things that are wrong. The agent caught it, too. So, I copy pasted into my code agent to go make the changes. Sure, there are some things on GitHub that have helped make this a little better. But wouldn't it be cool if the code agent actually had access to that directly and didn't have to bother you? Guess what? They did that. It's the CLI. It's really good. You should try it. Seriously, the code rabbit CLI has changed how I do these longer loop agentic coding things cuz I just tell the coding agent, keep running this command until it has nothing else to say and it will go for two to three times longer as it finds all these mistakes. Combine this with something like GPT 5.5 fast and you'll get crazy loops that can just go forever. There's a reason why everybody from Hura to clerk to bun is using code rabbit and it's not just because the reviews are good. It's because it understands your codebase. If it gets something wrong, you can tell it. If this affects something in a different repo, it knows that. It just gets how your team is working and it gets better over time as you tell it what it gets wrong. That's why my team loves it so much because we've taught it how we like to do code and it will just kind of enforce what we expect. You're not letting AI review your code, you're wasting so much time. Fix that now at soyv.link/codrabbit. GitHub got hacked. And to be clear, I'm not talking about the npm poning that's going on constantly. I'm also not talking about the CVE from two weeks ago where somebody found a way to do remote code execution on GitHub. In fact, their lack of address of this is so bad that I almost want to make a separate video about it. I might in the near future. They are not doing what npm needs and they are kind of trying to blame shift in a way I don't love. The npmjs account came back from the dead on Twitter to post about the supply chain attacks and how they want to prevent them from following patterns like the mini shy attacks. They invalidated all npm granular access tokens with right access that bypass two-factor authentication. You have to update your stored tokens and rerun the workflow for your automations. They also specify that you should be using trusted publishing with npm to reduce reliance on such tokens. This doesn't solve anything. I'm resisting the urge to go into a bunch of detail on this, but this is such a cope it is insulting. Most of the packages that were compromised were done through npm trusted publishing which if you're not familiar allows you to attach a specific GitHub repo in action to a deployment pipeline for npm. So the only way that can be deployed is through that action on that repo. The issue is it's possible to poison caches in GitHub actions if you don't use the exact right flag for your triggers on your GitHub actions, which is how things like Tanstack got pawned because they did pull request target and not pull request as the trigger for running their actions. Somebody maliciously filed a PR, immediately deleted it, triggering it to poison the cache, and then the next released restored that cache with the exploit. All of that happened within trusted publishing. So this doesn't solve [ __ ] anything. And it's insulting to the people who have these problems that are experiencing these things right now to imply trusted publishing fixes. And it makes me bearish on the state of GitHub as a whole right now. There's also a remote code thing a few weeks ago, but I've talked about that enough places. Surprised it doesn't get much coverage, but it is what it is. It got fixed before it was exploited as far as we know. None of that is the exploit we are talking about now where GitHub's internal everything seems to have been pawned and people got access to it. So their internal repos specifically I ended at the start that the reason they got pawned was another Microsoft thing and it wasn't npm. I was kind of banking on it being npm because it would have been funny that npm's horrible security practices right now would result in Microsoft itself getting pawned. But I made a grave error in this assumption because npm is not the surface that Microsoft owns that has userpublished software that is poorly reviewed and securely checked. There is one thing I know of that Microsoft does a way worse job of doing that type of due diligence with. I'll give you another hint. It's one of the apps I have open right now. It's VS Code. The VS Code extension marketplace is full of as much typo squatting and fake [ __ ] as it has actual packages in it. Many of these extensions are just straight up malicious. And I have talked about these many times before. The sheer volume of malicious extensions in the VS Code marketplace is genuinely absurd and not much has been done about it. They confirm this publicly with this thread here. We're sharing additional details regarding our investigation into unauthorized access to GitHub's internal repos. Yesterday, we detected and contained a compromise of an employee device involving a poisoned VS Code extension. We removed the malicious extension version, isolated the endpoint, and began incident response immediately. Our current assessment is that the activity involved exfiltration of GitHub internal repos. The attacker's current claims of 3,800 repos are directionally consistent with our investigation so far. We move quickly to reduce risk. Critical secrets were rotated yesterday and overnight with the highest impact credentials prioritized first. We continue to analyze logs, validate secret rotation, and monitor for any follow-on activity. We will take additional action as investigation warrants. We will publish a fuller report once the investigation is complete. Some of these replies are very telling and describe how absurd this is. I love this one from Darren. Just to be clear, Microsoft's GitHub was compromised when a Microsoft developer using Microsoft's VS Code installed the rogue extension from Microsoft's VS Code extension library, which is moderated and hosted by Microsoft. Back in 2024, the at the time CEO of GitHub posted about the co-pilot extension in thirdparty agents, specifically saying that Perplexity's agent was pretty cool and builtin. Someone here replied, "Can you fix the issue with people deploying malware in VS Code extension marketplace? I'm getting tired of sending mails every week to the hidden email that they had for it. Please fix your marketplace. I saw this post because Tay cited it in the replies. People have been begging to help Microsoft get their arms around this easily detectable [ __ ] in VS Code for years. And they are far from the only ones trying to do this. There's a company called Socket that you've almost certainly seen pop up in my content many times because they're often the ones who find these attacks first. They found the mini shy attack first. They keep finding all of the other similar things leaking out since then, including hundreds of packages in the AntV ecosystem getting compromised. It is my understanding they have a crazy analysis pipeline where they check updates to popular packages and then analyze them to see is there anything in here that might be malicious. They're very good at this. They help companies secure their supply chain for packages and the things that they consume. And they've done so well now, especially in this era of security becoming a bigger and bigger deal where they were just able to raise a lot of money at a much bigger val. Yeah, they did a 60 mil series C at a one bill valuation. This is actually very telling to have that low in amount at this high a val. Very rare. That is a 6% sale of the company. Not even. They did that because they had the power to do it. This means that they don't need the money. I know that seems crazy to raise 60 mil and say they don't need money, but they would have raised a lot more if they did. This is more of a a raise progression flex type thing where they're bringing in another lead, they're raising some more money and they are signaling to the world, we are ready to [ __ ] go. But the fact that this small company that now is admittedly worth a lot of money is so capable of detecting these exploits before anybody at Microsoft even notices and npm hasn't taken the time to give them a button they can click to auto remove [ __ ] If I was at npm or GitHub or Microsoft at all, I would have already gotten deeply in touch with these guys. And if you couldn't afford to acquire them, you should absolutely have given them a magic button they can hit whenever things are exploited because there is no way for them to do anything other than post about it and hope somebody at Microsoft sees it. And I'm tired of having to send 2 a.m. texts to my friends at GitHub in order to get like hijacked accounts unbanned and [ __ ] It's obnoxious. the the the severity of Microsoft's lack of security culture around these things is absurd and it's going to keep burning them aggressively. A similar firm named Nikito actually did a write up about this particular exploit that I think is worth taking a look at. Apparently the NX console VS Code extension also got exploited. did see the NX guys got poned really hard with a lot of the mini shihila stuff and once that pones you it gets access to all of your tokens all of your deploy methods and then that can be abused from there. Narwhal the creators of NX shared a bunch of info about this. A malicious version of the NX console was published at 12:30 p.m. UTC and removed soon after at 12:48 leaving it available for 18 minutes in the Visual Studio marketplace for Open VSX. The problem was detected later and the Comra version was available for 36 minutes instead. Lerna used to be the state-of-the-art here. It was a very popular solution Microsoft made for managing these things. And I'm sure it was used heavily at GitHub. Eventually, Microsoft stopped iterating on it and eventually let Narwhal take over maintenance. And my honest guess is that a lot of packages internally at Microsoft probably moved over to Narwhal at that point. I'd be surprised if that wasn't the case. I'm I'm not positive NX is used internally at Microsoft. So, it would not surprise me if this exploit happened through Narwhal exploitation specifically. We don't know for sure what extension it was. They haven't said publicly, but I would be surprised if it wasn't this. But Aikido's reporting is great. So, let's read more about this. That extension had 2.2 mill installs as well as a verified publisher badge. And the compromised version was uploaded on May 18th at 12:30 UTC. Microsoft did not flag the upload. The NX maintainer did not receive the marketplace's upload notification email until 6 minutes later. The compromised build was unpublished 10 minutes later and Microsoft registered the takedown. As they said, took about 18 minutes. We already said the Open VSX thing. Here's where things get fun. According to the advisory, anybody who had NX console installed with auto update enabled during that window should assume they were compromised and anything on disk, tokens, secrets, SSH keys, anything credentials adjacent should be rotated. The root cause was a contributor's GitHub token that had been scraped in an earlier supply chain attack and then used to publish the malicious release. This is possibly the scariest part here. Many don't understand the severity of all the attacks that have been happening over the last few weeks because of the nature of the Shyellet attack compromising so many packages. They have too much data to go through. Whoever did that hack is sitting on this giant pile of data. They're trying to get through it. And every few days, they're going to find something they didn't realize they had. They're going to test it and confirm that it's a token that still works. And then they're going to go do a whole new set of exploits because they have effectively collected all of these passwords for all of these different things they shouldn't have and it's too much for them to even go through. So they're still going through them. Even if all the security [ __ ] going on right now gets patched, those tokens are still held by a bad faith actor that will continue to exploit constantly. And since that token, it says that token was grabbed by these malicious people for publishing this extension, it didn't take much effort to figure out what it was and then attack it. Historically, going through all of this data and finding all of these angles to attack would be months of very carefully done human work. I would be beyond surprised if they didn't have an agent just running in a loop over all this data, trying to find things within it that it could use for further exploitations and then automating it from there. Previously, it would have taken at least a little bit of effort to make a fake release of this extension just to add in your malware and publish it. Now, you just tell an agent, "Hey, find the GitHub repo. How do I publish this? Here's a token. Publish this version." And it can do all of that. A big part of why these things keep happening is auto update, which is crazy because previously updating was the way you stayed secure. Now that all these tokens have leaked, updating is a little insecure, which is just insane. Every popular extension marketplace ships with auto update on by default. VS Code, Cursor, the whole lineup. I'll go a step further and say Chrome was an underrated thing for this, too. Most of the attempts to pone me have not been because people want my YouTube or my bank accounts or access to my data. They've been because I made a popular Chrome extension over 10 years ago. And people like millions of installations of that extension exist. And a lot of hackers have been trying hard to get into my Google account so they can publish malware via that popular extension. Someone actually burned a novel twof workound exploit where they got through my two-factor authentication without me entering anything in order to publish without needing it in order to get around that to get to my users and install malware. It's insane. Extension auto updates are a massive attack surface. The reasoning for these auto updates does make sense in isolation because devs never update anything manually. Users do even less. So I had to call my parents and tell them they need to do their iOS updates now. been sitting and not doing it forever. They finally fixed it, but it's insane. And leaving off auto updates means a long tale of editors running stale and vulnerable code. The trade-off stops making sense once you account for hostile and compromised publishers. Though, auto update gives an attacker who controls a release a direct push channel into every machine running the extension. Marketplaces don't impose any review gate or waiting period between when an update is published and when installed clients pull it in. Yep, you'd think Microsoft of all places would understand this. The check itself doesn't have a single fixed interval. There's a 12-hour fallback timer in the extension workbench service, 1 hour on insiders apparently, and then an immediate check on startup once the worksbench is idle. So, if you close and open up VS Code isn't installs. There's also random things that trigger updates as well, like the allow list, policy changes, product update checks, etc., there's also a path that will auto update and find these updates whenever you interface with the marketplace at all. When VS Code queries the gallery for any reason like extension sidebar visible, marketplace search, recommendation prompt, background, sub subsystems, etc., it syncs the result against installed extensions, and if any are outdated, it will eventually autoupdate extensions, which installs through a delay or throttled to 1 second. A lot of people got pawned by this. The 18-minute window in the VS marketplace caught any running editor that did a gallery interaction touching the NX console list during those minutes. Like if I have an extension here that's compromised, all I have to do to get the update that's compromised is click that. Once the extension sidebar is opened, it's triggering the auto updates. You might have noticed before I closed it that it showed a bunch of things had updates. Then I closed it opened and it no longer shows the updates because it just went and installed them. Across time zones and working hours on an extension with 2.2 million installs and extension sidebar that many devs leave open, the population is much larger than a periodic timer alone would suggest. During our own testing of these features, we've seen auto updates firing within minutes of publishing new versions, which matches the gallery sync path rather than 12-h hour timer. The email delay is also absurd that the maintainer didn't get notified till 6 minutes after during which people were getting this malicious update. So, why does this keep working? The NX case is not the first time this has happened to a VS Code extension. In November of 2025, the Async API's compromise was the first blood event of the Shyud 20 worm wave. The attackers had stolen both async APIs, npm and openvsx publishing tokens which have been sitting in a GitHub repo secrets for years. GitHub repo secrets should be safe, but if you manage to get an action to run maliciously with access to those secrets, you can steal them, which is what happened there. And they use those tokens to publish malicious async API npm packages and a malicious version of the async API vs code extension as well on OpenVSX. At least one developer reported being infected through the extension on the project's own issue tracker before maintainers could deprecate the affected releases. The same mechanism was at work. Stolen credential used to push malicious code to an autoupdating user base over a short window. Another big part of why this can work is the fact that you have infinite distribution. But more importantly, these extensions often ship as interpreted JavaScript rather than compiled binaries. But they'll minify and obuscate the [ __ ] out of that. And while sometimes Microsoft notices this in bands, they do it very very late. Maintainer credentials are easier to fish or scrape than they should be as well, which is part of why this can all happen. And the community is getting fast at catching. But 18 minutes of auto updates and auto distribution is definitely enough to do damage. And once it's on a machine, the marketplace can't really help you at all. Like, if you're hacked, you're hacked. If they change or uninstall the extension automatically, you're still hacked. Apparently, with the Shil 2 attack and the Async API vs code extension, people were still being pawned up to a month later because it was still running in the background. And that resulted in roughly 100 to 200 new compromised repos per day from November 25th when the attack happened through December 24th. The flow only stopped when Async API published a clean version which forced things to update to this new thing cuz 10 was the right version. The malicious one was 101. So when it got deleted, a lot of people still had 101 installed. They had to ship a new version in order to force people up to a clean one. Like what the [ __ ] I think we're nearing the point where we have to do [ __ ] very differently here. Just some ideas. And this is going to talk about the MPM [ __ ] more. I didn't want to, but I feel like we have to. I'll emphasize here more than ever, I'm not a security researcher. I just have slowly become security psychosist by all of this [ __ ] This is things I think would help. Not things that would solve the problem, just things that would help. First and foremost at a threshold packages need to be analyzed on update automatically by npm GitHub etc. So if you have an extension in VS Code marketplace that has more than a thousand installs or an npm package that has more than a thousand lifetime downloads or is depended on by other things that have really high download rates, you need to have a system in place that will audit those updates when they come in. There's no reason startups like Socket and Aikido should be going so much harder on this and doing so much better at this than Microsoft who has the incentive to burn the tokens. Throw an agent in on the updates of these important things and do a quick audit pass before you actually trigger the installations across the market. In order for this to work, one of the other things you need is a staging flow. A new update should not be automatically installed when you do like npm install package or npm update until this check has passed period. You can still manually choose to install the latest version during that window. But npm itself should not be autoinstalling the latest version if it's only a few minutes old and hasn't been audited yet. The most important piece here though is the takedown flow. There is nothing here. For a long time, npm's policy on this is that they'll never revert a package because somebody might be depending on it. This has been annoying as a developer just building on npm accidentally shipping the wrong version. There is no way to undo that. It is impossible to unhip a package to npm. That is just nonsense, especially now. I get why they wanted to do that. You should be able to reverse a publish if it's been an hour since you published, no matter who the [ __ ] you are. And when you do that, it should send out some type of push to the developers who have installed that version. So I can check a box that says this is because it was malware and then something gets pushed out in some way. The method of this is tough to get right. The best thing I can think of here is some optional environment variable you can attach to your npm in your npm runners inside of your GitHub actions and whatnot that has an email associated with it on the npm dashboard. And when a malicious install occurs or any install occurs that gets logged as like this hash installed this thing and then when a reversion happens because a malicious takeover occurred, you go through all the people who have that installed and then push an email to them letting them know, hey, this got [ __ ] up. Fix it. And maybe you even keep a registry of these versions that were compromised. And every time you run an npm command, it'll check in the background against this registry and give you a big alert in your UI or CLI saying, "Hey, just so you know, this is really fucked." Yeah, like something drastic needs to happen. There is no combination of the things npm and the VS Code marketplace already do that will solve this problem. And there isn't really a way to secure yourself other than spending a ton of money on services like socket. The solution needs to be rethinking these core pieces that npm and the VS Code marketplace have tried to convince us are correct and they're just not like obviously. So it's time to be willing to rebuild these things. It's time to give up these existing strong opinions and points because they are going to result in more and more compromises happening. How many fewer people would have been compromised by this extension exploit if first off the notification went to the developer earlier? If second off there was a button in a dashboard they could click that said help this was a hacked release. If third off they had some staging process where the updates didn't get autoinstalled until a certain amount of time had passed like a safe buffer window. And most importantly there was a process to push out an unupdate saying uninstall this thing go back to the old version. Any one of those things would have massively reduced the number of people affected. The entirety of those things together would have 100% prevented this. But Microsoft doesn't seem interested in changing these things because they still don't feel like they have to. They have to. Now, it was just confirmed by the CEO of Narwhal, by the way, that the GitHub compromise was indeed the NX console extension. It's a difficult thing to read as the CEO of NX, and I want to be direct about it. We take responsibility for a role the software played in the incident. I don't think it was that much, you guys. Like I I think Jeff's taking too much responsibility here, but I'll read what he has to say. I'm grateful to the GitHub, Microsoft, and independent security teams that move quickly to investigate, contain, and share information publicly. This incident highlights that there needs to be deeper, more fundamental changes to how we and other maintainers need to think about securing developer tools and open source distribution. We are already making major changes to our publishing, automation, and extension security posture, and we'll continue sharing those changes publicly as we implement them. also beginning convos with other high-profile open source maintainers about how we can work together on some of the deeper structural problems around software supply chain security. A lot of the assumptions the ecosystem had operated under for years no longer hold. Yep. And I will continue to point fingers at npm, GitHub, and Microsoft for doing [ __ ] nothing. They have not given us the tools we need to do this right. It is not our responsibility to reinvent the universe because the core that we're building on top of npm has been a rotting [ __ ] corpse for a decade now. NPM was so far ahead when it dropped that they haven't felt the need to fix things since and it shows now more than ever. The rough terrible attempts to do this right through like the trusted publishing flows and [ __ ] are so obviously not far enough that they are being used as part of these compromises. Now, it's time for them to wake the [ __ ] up and fix this. NX's focus right now is supporting affected users, hardening NX, and helping push the broader ecosystem towards stronger supply chain security practices. This is not their fault. I am going to keep yelling at Microsoft about this. Both because their systems are too easy to compromise, but more importantly, hackers have too much incentive because Microsoft hasn't done jack [ __ ] [ __ ] to disincentivize it. If they were to add a 12-h hour delay for compromised extensions before the auto updates would start happening, there would have been no incentive for the hacker to bother doing this. A very simple safety window applied by Microsoft to these things combined with auto analysis, like very basic, run a cheap agent against the changes and see if there's anything vaguely suspicious. That combo isn't much work and would literally save thousands of developers from being compromised and potentially millions of users as well. It's time for Microsoft to wake the [ __ ] up to the reality they've put us in by not addressing these things early. They have to do something because all of these maintainers hands are tied. And also, to be frank, these open source maintainers already have enough [ __ ] on their plate. If they have to worry about this as well, that's it's just not [ __ ] fair. people building free and open source software having to worry about getting SIM swapped or their machine having an npm update that it shouldn't have causing all of their tokens to be compromised and then pwning their millions of users is all Microsoft's platform. This is all an arena Microsoft owns and we're trying to figure out how to safely navigate it because Microsoft keeps accidentally lighting fires inside. Can we do something about this? Maybe. Is it our responsibility to? [ __ ] no. And as much as I respect the open source maintainers who are trying to find solutions here, I want to make sure that their attempts to do this well and in good faith, do not take the crosshairs off of Microsoft and their [ __ ] They need to fix this. A stupid [ __ ] JavaScript YouTuber should not be able to see a path here and Microsoft just sits there with their hands under their ass. Like, what's going on? I got nothing else on this one. Wake the [ __ ] up, Microsoft. You're letting your users get hurt and you're letting the open source community falter as a result of your [ __ ] This goes way beyond the slow death of GitHub. This is going to kill all of software if you don't fix it. Get on your [ __ ] and deal with it, Microsoft. This is not acceptable anymore. Until next time, peace nerds.