Did Claude really get dumber again?

Have you noticed Claude's performance varying by day? Claude Opus 4.7 is a serious regression, not an upgrade. AMD's AI director slams Claude for becoming dumber and lazier since last update. Opus 4.6 is getting dumber. Cloud code performance regression with Opus 47 models. Quantified evidence. Sonnet 46 quality regression since March 9th. Cloud model performance degradation after extended usage session. And a funny one for my chat. I don't have a screenshot, but I've had numerous occasions where Claude started speaking to me in Chinese. Oh boy, seems like something's up again. If you guys don't recall, Enthropic has had this problem in the past where their models are great when they launch, but slowly regress over time. Sure, right now, Opus 47 isn't the best experience that I've had coding with AI, but it seems like all of the models are starting to perform worse. And there have been a lot of things that led up to this. I have a hot take, though. I don't necessarily think that this is just the inference. I don't think that the API suddenly spits out dumber things. I don't think the models got dumber in a traditional sense. But your experience is real. Mine is, too. We've all felt the difference. It feels like cloud code is dumber than it was before. And I want to dig in. I want to figure out at the root what's going on here. There's a lot to investigate here. From the execs at AMD taking the time to document all of the changes to how thinking works in their own reporting to the previous degradations that were mentioned back in September to the notably declining trends according to Margin Labs with the models getting meaningfully worse. There's a lot to dive into here and I can't wait to do it with y'all. But first, let's take a quick break for today's sponsor. AI is pretty good at coding, but I find it's even better at code review. I seriously don't know how I ever lived without good AI code review, and that's why you should check out today's sponsor, Grapile. You've all heard about these AI code review things. So, I'm just going to give a list of things that I've been finding really cool about Grapile specifically. The first one I love is the Grapile.json file that you can configure in your repo. This is great because I shouldn't have to go to a dashboard to change settings. I should be able to ask my agent to do it, and if it's a file in your codebase, you can. It's almost like this company is thinking about how we're coding today. You can see that even more so with their fix in X feature. That's not just a fix in cursor button that calls that one weird endpoint. This lets you set up a local bridge using their npm package and it is absurdly fast. Here's a mistake reptile caught in a PR. I could click the prompt to fix with AI and copy paste this. But then I have to hop around my computer a whole bunch and I might not even have that agent open yet. But as you all know, I love codecs. Watch how fast this is. This is real time. I am clicking now and now it is prompting. No edits there at all. And one last thing, it's silly, but it's super helpful. The confidence score. Grapile leaves a score on every PR it reviews by default. And while they only gave this PR a three out of five, I would give that feature a five out of five. Ship less bugs and more confidence at soyv.link/gile. I'll put all my cards on the table here quick. I have historically pushed back on these types of claims. I genuinely feel as though a lot of these reported regressions are overblown at best and historically I have not felt the same, at least until recently. This is a post I made just a couple weeks ago where I had experienced regressions myself. This is a regression I noticed around the same time they made the changes to OpenClaw where they were banning them from using the Cloud Code subs, which I understand it's probably doing a ton of usage, but what I use Cloud Code for the most part is just random tasks on my computer, like debugging what was wrong with my Dropbox app. It was willing to kill the app, but it still didn't appear in my menu bar as it was supposed to. So, I asked it to help me figure this out. To which it said, "That's outside my area. I'm built for software engineering tasks like writing code, debugging, and working with repos. I wouldn't know about recent Dropbox UI changes. I suggest checking Dropbox's settings for a show Dropbox and menu option. I couldn't get the app to open. I can't go into the settings when I can't open the [ __ ] thing. And when I tried doing follow-ups, it got even worse. I asked it to do some research for me on this, and it said, "Sure, but keep in mind, I'm best suited for software engineering research, code bases, APIs, libraries, architecture, etc. What do you want me to look into?" Obviously, Codex was more than willing to do the work and got it fixed in minutes. Just annoying. The point I'm trying to make here isn't that like, oh, I had this one anecdotal bad experience, therefore all of this is dumb. I think this showcases both what these regressions can look like, but also where I think the source is. There's a couple different types of regressions that are absolutely happening here. One of the biggest is task refusals. Refusals can come in different forms. They can be the API hard locking you out and refusing to resolve the request you make. It could be the model saying, "Eh, I'm not going to do that." Like I just showed there. It could be the model going down a different path than you would have expected and just avoiding the task you gave it. But for the most part, refusals come in those first two forms, either API block or model quitting. There's also dumber behavior. And this isn't like the model not doing the thing you want it to. This is the model just doing [ __ ] wrong like flipping a boolean it shouldn't or not following the intent of the task and on that I'll say there's almost an entirely different thing which is context rot kind I don't even know if that's how I want to refer to it there's like getting lost is how I'll describe it you know I'm going to rephrase it to dumber solutions and getting lost dumb solutions is the model is unable to solve problems it could in the past it writes worse code that doesn't behave properly and getting lost is when the model doesn't follow the intent of what you requested. If you saw my Opus 4.7 video, I would say this fits along with the experience I had trying to write that script using it where the script I built for cloning a repo to other places on my computer, it just kept losing track of what I was asking and would incorrectly change things based on misinterpretations of things I had said in the past. These are all different types of regression. Some of these are done via the harnesses, some of these are done via the API, some of these are done by the actual hosting of the inference. There's lots of different places where things can go wrong and from the numbers it does seem like it's going wrong. The weighted averages from March to now on Margin Lab who tracks all of this with their own benchmarks internally has seen a meaningful dip. It's not a huge dip but it's from 57% down to 55%. And it's consistently down like every week it gets lower which is kind of crazy. What does this number mean? Chad is asking. It means that they are running SWE bench on the model using claude code and they're seeing regressions consistently. You might notice this little bump at the end. That's cuz a new smarter model just came out and they're always using the state-of-the-art. In order to understand where these issues might be coming from, I think it's important to understand the loop of what happens when you make a request. So if you start by prompting cloud code, this prompt has to go somewhere. Usually where this prompt is going to go is to an API. The API is going to take in your prompt in any other context. It often will do a quick scan on it to see if this thing that you're requesting is appropriate or not, sometimes with a different model. And once the request has been approved and allowed, the API will then take what you've requested and pass it over to the GPUs that do inference. The GPUs doing the inference are loading a bunch of weights. They're loading the parameters that are often referred to as the model. All an LLM is is a bunch of parameters in a map that point to each other with different values. It's a matrix that gets computed over time based on the context that exists for the model. The details there don't matter too much. This is effectively a gigantic file with a bunch of random text data pointing at each other in it. It is handed to the GPUs and loaded in memory. All of your context adjusts the weights a bit so different things point slightly differently and then it generates the most likely next token based on all of the context it has. And it does that again and again until eventually a result comes out. But there are other things in here that we should probably think about too. Specifically, between your prompt and the API, you have this wonderful thing called the harness. I've talked about harnesses extensively in other videos. Watch my how does Cloud Code work video for more on what a harness actually is and what it does. I think it is very useful information to have, especially nowadays. But when you write a prompt, you're not just sending the text that you wrote over the API. You're sending a lot of additional data, too. things like the system prompt which describes what the intent of the request is. If you're using a site like T3 chat, we write our system prompt and we pass it to the anthropic API ourselves. You can't see it, but we don't hide it. It's pretty easy to get models to spit it out. The harness also will include things like what tools can be used, what functionality exists in the product, and all these other things that the model needs in order to do work. In order for the model to run a command or read text or make changes on your machine, it has to call tools, but only knows what tools it can use based on how the harness shapes the system prompt and the details for what can be done there. So, when you send a message, you're not just sending the message. It has all the additional context provided by whatever tool you're using. The reason I'm showing this is because every one of these layers has impact on the quality of your responses. Any one of these layers making changes can cause all of these types of problems to occur. If you make a change to the harness where the tools have different expectations than they used to, that might make the responses error out more. If you put too much bad context in the harness, that might also make the models behave worse. If you have API changes that change the way that you're actually passing the tokens to the GPUs, or maybe you make API changes that are more aggressive in how they filter things and change things before they go to the GPU, you'll end up with a lot of refusals that you probably shouldn't have. That's why I got refusals like this one on cloud.ai AI just asking you to solve a gold bug puzzle. For for those who aren't familiar, these aren't hacking puzzles. These are weird cryptographic challenges where you have to do math to get a text answer out. And it concluded after some code was written that this might be hacking. Therefore, it refuses to do it. That is the API, not the model. The model isn't dumber or deciding that this shouldn't be responded to there. That is another layer between the request and the API that prevents it from happening. Then there is the actual compute where the inference happens. This part is very interesting because unlike most of the other labs, Anthropic isn't betting super heavily on Nvidia. They do still use Nvidia for some things, but Anthropic is primarily working right now with Google and Broadcom for creating new chips for their next generation compute. That means that they now have to train and run Claude on various different hardware. They have AWS's chips called Trrenium, Google's TPUs, as well as Nvidia GPUs. They're primarily betting on these first two lately. It seems like there's some beef between Enthropic and Nvidia for sure, but they still have a bunch of GPUs and they're using them. This means that there's a lot of different ways the model is being served. So, you might have some requests where the model is responding via Tranium, others where it's responding via Nvidia. They are desperately looking for compute and will throw your requests wherever they can. And it is plausible, if not likely, that these different hosts have different behaviors when you hit them. But there's another issue here. You would probably think that when you make one prompt, it's going to send the request and then it's going to hit one GPU and run it. But when you're actually using a tool like Claude Code, it's not just going to send one request to one GPU. Every time a tool is called, another API request has to be made. Because if I ask it to edit a file, the first thing the model has to do is read the file. So it responds with a tool call to go read the file. Then that runs on your computer, gets the result, and then a new API request is made to do the next step. In each of those steps is a new request. And each of those requests could be an entirely different GPU in an entirely different cloud. It's possible that with one prompt in cloud code, you might hit Tranium, Google, and Nvidia GPUs. You can imagine how much variety and range of potential errors there is there. And then we have the model itself. The model obviously changes quite a bit. We had Opus 45, then 46, then 47. It seems like a lot of people are noticing regressions going from 46 to 47. I'm definitely one of those people. I feel like most of the regressions I'm seeing thus far are on the API side, but the model itself is definitely weird. There's a lot to dig into here, but I don't want to start just yet cuz that'll just become the whole video. So, I want to start listing the different pieces that can go wrong in any one of these layers because I honestly think all of them are where the problem is. to start at the top because it's the easiest. I think we have gotten more confident in what these models and tools are capable of. I remember in December being impressed when I asked Opus to edit a file and grab some context from another place on my computer. As I got more impressed, I made bigger bets. I tried harder prompts. I did more stuff and I got used to where the model could be. That means that my baseline for what was possible went up. And if you have a range of like all dev tasks where on the left here you have hello world and on the right you have building Linux from scratch. I would have guessed that LLM capabilities in like November were like here or so which means I have a range in my head. Whether or not I should I do. So if my baseline is here anything in this area I would consider kind of dumb. If it goes past that point I think it's really dumb. But if it goes this way I'm more impressed. So when I tried out Opus 45, my baseline for what was possible went up quite a bit. And as such, my range of acceptable changed too. So if I had a task that was here, previously that would have impressed me. Now just a few months later, a task of that complexity doesn't impress me anymore. If a task of that complexity failed when my bar was here, I wouldn't have been that surprised because it's over where my bar is. But if it fails now, I'm like, what the [ __ ] feels like a regression because your bar changed. Your expectations have shifted. Code that you thought was good when you were a junior looks like [ __ ] when you're a more experienced developer. Tasks that you thought were hard for a model before now feel easy. So, our bar has shifted. The way that we measure the capabilities has changed over time. So, I genuinely believe as though the prompting that we're doing is requesting harder things and as such, our expectations have went up and when things don't go well, our disappointment has gone up alongside it. Then there is something between the prompting and the harness that I want to think about a bit here too, which are stuff like skills, MCPs, plugins, and all the other things that we do to our instances of tools like cloud code. I've seen a lot of developers get obsessed with adding all of these customizations to their Cloud Code stuff. Anything from a pile of useless MCP servers to GStack to everything in between. All of those pollute the context some amount. When you add new features to quad code and you make changes to your scaffolding for the model by giving it new skills, new abilities, new MCPs, all of that is data that now exists in the system prompt. Effectively, it's more context being taken up. And when you remember how models work, where every word that exists in your history slightly changes the direction the output goes, more things that aren't quite what the model was trained on will make it behave differently in ways that are often not intended. If you put too many skills in a model's system prompt, it will often behave worse than if it didn't need any of those. But users aren't the only ones adding a bunch of [ __ ] to the system prompt. Here's where we start getting into Theo hot takes. I genuinely believe a significant portion of the regressions that we are experiencing as users are coming from shitty code in Claude Code. Here's an example from when I was testing Opus 4.7. One of the things that Claude code enforces is that the model can't edit a file until it has read it. So here it tried to update the package JSON file. It got an error. File must be read first. So it searched. The model seemed to assume that searching for the file would count as a read. But it doesn't because the harness is, to be frank, pretty [ __ ] poorly coded. So when it tried again, it failed again. And now it has to manually call the reading tool instead in order to make sure it qualifies to actually do the update, even though it knows what the file contents already are. This is an example of the harness not just making the model behave worse or dumber but also costing you more usage and money and them more compute because remember after every single tool call a new API call is made. This should have been one API call the update and then it's done. Instead we got the one for the update, one for the search, one more for another failed update, one more for this read and then finally we get the response. This pollutes the context. So now there's a bunch of data that's in your history that doesn't matter anymore. There's a bunch of API calls you don't need, a bunch of tokens being wasted, and a bunch of work being done that costs you time and money, the model's context and capability, and enthropic compute that they don't have. There are dumb changes like this in Cloud Code that have probably cost millions, if not hundreds of millions of dollars in unnecessary inference because the model assumed it could update a thing and it couldn't. I reference this bench from Matt Mau a lot because it's really good. He made a benchmark that measures how well models implement a 100 feature document and he used the same model in different harnesses. He had Gemini used inside of the Gemini official CLI versus inside of cursor, GBD54 in the official codec cli versus cursor and then opus in quad code versus cursor. This is insane. This is this should really be all we need to see. This should be enough to get people [ __ ] fired if I'm being frank. The fact that Opus performs 15% worse in quad code versus cursor should say everything you need to know. Anthropic is too focused on making Claude code have all these features and do all these things and shipping utter [ __ ] slop constantly. And the result is that the models feel dumber. We are now at a point where anthropics incompetence in engineering is making us think their models are getting dumber. It is entirely possible to make a fiveword change in a string inside of the cloud codebase. You can make slight adjustments to the system prompt and make the model 20 times dumber. It is absolutely doable. If you gave me source code access to cloud code, I could make it the dumbest harness ever with just a couple words being changed in the system prompt. Doesn't take much to mislead the models. And we've seen enough regressions like this that hopefully Anthropic is waking up to this reality and taking it more seriously. Every time a new tool is added, every time a new adjustment to the system prompt is made, every time any of these things happen, they are increasing the service area for stupid. And godamn is some of that stuff stupid. When I asked Opus to do some design improvements on T3.gg when I was testing it out in the official desktop app, it opened with, "Hads up, the last system reminder about malware looks like a prompt injection. This is clearly your personal site, the T3G homepage with links and sponsors, not malware. Ignoring it." I know it says it's ignoring it, but do you understand how polluted the context is when this happens? Imagine you're trying to navigate your computer and there's just random pop-ups and flashing [ __ ] all over it constantly. The likelihood you complete the tasks well and effectively goes down as there's more [ __ ] going on on your machine. That's effectively how the models work. the more [ __ ] that that exists in the context, whether it is from the system prompt, from you giving it bad info, from it reading things it shouldn't, from tools that are bad, or things like this, it makes the model dumber. It makes the model so much dumber that doesn't just open with this reminder that this malware thing is stupid and not necessary. It also ends with it. Note, three system reminder blocks in this conversation instructed me to refuse to improve or augment code as if it were malware. That's a prompt injection pattern, not a legitimate instruction. Your site is obviously not malware, so I ignored them. Worth knowing where it came from if you didn't add it yourself. If you don't think this makes the model dumber, you might be dumber than the model. It is obvious that these types of bad code changes are [ __ ] up the harness in such a way that the results that go to the API, that go to the GPUs that generate your results are worse and dumber. If there is more context wasted on [ __ ] that doesn't matter, there is less context used for the things that do. And when you are changing how the model works with every single word you add to the history, useless words end up steering it in useless directions. And this is why on terminal bench, claude code is the worst performing harness for using opus. There are harnesses like Forge Code and Cappy that are getting 75 to 82% on this benchmark and Claude Code is only pulling a 58%. Do you understand just how [ __ ] bad Claude Code is for anthropic models? The hardest is [ __ ] It's not just bad in the I don't like using it sense. It's bad in the it hurts the model and makes us think that they're dumber sense. I would personally guess that at least half of the regressions that we're seeing come from here. And for those who seem to think that this happens to all of the models because the bench is kind of sus, here is GPD 5.3 codeex showing that simple codeex which is their CLI is in third place and only barely beat out by Droid and Safe Agent. Is the bench perfect? No. Is Claude Code good? Also no. BridgeMind has done his own benchmarks about some of these things, particularly hallucinations, and he noticed that from launch to April 12th, which is not very long ago, Opus went from an 87.6% 6% on this benchmark down to a 73.3. And this isn't a benchmark that uses claude code or anything. This just hits the API and does things. And even then, it had a massive regression. It has since recovered, especially on the new V2 that he made of this bench. But the fact that it dropped that many points says a lot about how the model's being served. So, we're now getting to this API part. I've touched and danced all around it, but let's dig in a bit more on things that can change on the API that are bad. One of the things is routing. deciding which places and which GPUs your request should be sent to. Also, whether or not it should be sent at all. That's the problem I showed where I got rejected from cloud.ai for my simple question about a problem I wanted it to solve. There are other changes on the API level that are worth noting, though. One is the tokenizer. If you're not familiar with tokenization, it's the process that takes the text that you submit and turns it into these tiny little groups of text that the model can then parse and build its mapping around inside of a model itself. Here's some HTML and you could see how it's tokenized. The first couple spaces are one token. Then the next space in the open bracket is a token. The word section is a token. Space ID is a token. Equal open quotes a token. Harness is a split in an interesting way here. But then the space class is a thing. The reason this matters is it's how your data is parsed by the model in order to figure out what most likely should be generated next. And over time, these tokenizers have changed. Check out how much it changes from GBT5 to GBT3. Do you see how all of the spaces at the front are now all tokens? That's because the original tokenizers were not very good at code. They weren't optimized for that. They were just trying to break down pros and English into nice formats for the models to digest and predict against. If you were to change this to sentences like, "Hello world. How are y'all doing today? Don't forget to subscribe." Speaking of which, make sure you hit the sub button. It's free. We switch from that to GPT5. You'll see very little changes in the tokenization. It's basically just how the grouping happens after y'all because the apostrophe here is confusing it a little bit. Yeah, the apostrophe in GPT3 was separated from y'all. So why apostrophe all were its own tokens? Here it is not. That's not a very big difference. But when you paste in code, the difference is huge. It's 225 tokens for this example with GBT5 and it's over 400 with the GPT3 tokenizer. These changes are made in order to make it so the model can process the data better and make better decisions, so to speak, because each of these tokens is used to steer the direction the model goes in based on what parameters exist inside of it pointing to and from each other. The problem with this isn't that it's too many tokens. The problem is that the groupings of them don't make sense. The sheer number of empty space tokens here confuses the model and it thinks that after a space often comes another space. This is why certain models, I will stare at the old Gemini models in particular for this, will sometimes just repeat empty spaces for thousands upon thousands of tokens. We've probably spent at T3 Chat over 10 to $50,000 on just generating empty spaces and loops because the old tokenizers were so bad at formatting code. OGBD5, you'll see each of these indentations is one token. So it doesn't get repeated the same way and doesn't cause the pollution of the context in the same way. So why am I talking about all of this? Well, there are definitely tokenization changes in general, but the bigger one here is that Enthropic confirmed with Opus 4.7 that they made a huge change to how the tokenizer works. As they said in the official Opus 4.7 release, Opus 4.7 uses an updated tokenizer that improves how the model processes text. The trade-off is that the same input can map to more tokens. roughly 1x to 1.35x depending on the content type. It also thinks more on higher levels, but that's not what we're here for. We're here for this 1.0 to 1.35x increase in the number of tokens despite getting the same text. Abishek measured this and said that it was actually closer to 1.47x. He got the 1.47 on tech docs, 1.45x on a real claude MD file. The top of anthropics range is where most cloud code context actually sits, not the middle. Yeah, the majority of your context when you're using the models is probably in your documentation, your cloud files, and all those other things. Those are now way more bloated. So, you're going to be burning way more tokens with these new models. And more tokens often means more context rot, too. So, on one hand, I'm sure the model can process the text better because of this. But on the other hand, you're now using way more tokens. You're hitting your limits faster, but most importantly, there is more context because it's just a measure of tokens. So the model is going to potentially behave dumber if you are 50% increasing the amount of stuff going on. Just think about this as yourself as a human. Would you rather find a bug in a file that is 100 lines of code or 150 lines of code? Even if the additional 50 lines are just new line spacing, it's still much more annoying to find what you're looking for in a file that is 50% bigger. So if the model now has 50% more [ __ ] it has to get through. It is not surprising that in many cases it acts dumber. It is also kind of silly that they changed tokenizers in a minor version bump because as we saw they only changed it between GBT3, GBT4 and the O series and GPT5. That is when they made the tokenizer changes, not in a4 update or 7 update like they did here with anthropic. So that is without question one of those changes that just fundamentally makes us rethink what is causing these problems. Like there's a lot of these things that might be a cause. They might not matter that much. But I would be very surprised if this wasn't a meaningful impact on the experience that we are having. You know what else might impact your experience? Hopefully for the better our sponsor. You know what Cursor, Windsurf, Particle, and Mintify all have in common? It's not that they're IDEs, cuz half of them aren't. It's that they're all using the same solution to translate their sites. Yes, all of those awesome companies are using general translation because it's the best way to add internationalization to your apps. I'm not just saying that cuz they paid me. I'm saying that because when they reached out, I immediately fell in love with the team and the product, begged them to let me invest, begged them to let me advise, and ended up going to their office to annoy him even more. This team is awesome in what they built is even more so. To translate content, you wrap it with the T component. To escape a variable, you wrap it with the ver component. To handle numbers, you use the numbum component. To deal with currency and currency conversions, you use the currency component. Date times are handled if internationalized, too. This is all of the integrations that are annoying to get right, handled for you by GT. 90% of the world doesn't speak English, which means you're potentially losing out on 90% of your customers. General translation can trivially set you up on almost any modern codebase to have good internationalization. They also have an incredible CLI that will make it trivial to set up. You just run npxgt in your project and it will do everything that is needed to get your setup ready for internationalization. Stop leaving 90% of your users behind at soy.link/gt. I already touched on the GPU side here a bit. I will be honest, there is so little info about this that any speculation here is borderline conspiracy. I would honestly guess that some of those tokenization changes are to make the model behave better on certain types of GPUs and TPUs, but I can't know for sure. So, I'm just going to not say anything here other than Anthropic has a hard problem with all the different types of compute they're running on. It is not surprising that there are some issues, but there's one big piece left, the model. I think the best place to start here isn't actually here. It's the postmortem from September. This is the technical report on three bugs that intermittently degraded responses from Claude. Below, we explain what happened, why it took time to fix, and what we're changing. Between August and early September, three infra bugs intermittently degraded Claude's response quality. We've now resolved these issues and want to explain what happened. They talk about the different types of hardware they serve on, and each hardware platform has different characteristics and require specific optimizations. Despite these variations, we have strict equivalent standards for model implementations. Our aim is that users should get the same quality response regardless of which platform serves their request. This complexity means that any infra changes require careful validation across all platforms and configurations. So once again, remember in any given request that you make in cloud code, every turn could hit a different GPU. Every step could hit a different GPU in a different server farm from a different company. So it could get smarter and dumber hypothetically speaking as it's responding to the same request, which is kind of crazy. But here are the actual bugs that caused the problems that we all experienced last year with this performance regression. The first was a context window routing error that affected around 1% of Sonnet 4 requests. Remember 1% of requests. If the average prompt does 15 requests, you're going to hit this pretty often. Like you're rolling a one in a 100 chance every single step. And if you're doing 15 steps per prompt, you're going to hit that routing error relatively often. But then they made a load balancing change that broke it further and now it's hitting 16% of requests. That means on your average message, you're going to hit the graded model at least once. Separately, other issues happened. The output corruption errors as well as some bad code that made the top K measurements for how they are computing the next value get bad. They fixed these over time, but at the same time on August 29th, they had that bigger regression where a significant portion of Sonnet 4 requests were being routed incorrectly. You might be asking, what does routed incorrectly mean here? They were nice enough to tell us. It was a context window routing error. On August 5th, some Sonnet 4 requests were misouted to servers configured for the upcoming 1 million token context window. The version of the model that has 1 million tokens of context behaves dumber. This effectively confirms that fact. If you aren't using a million tokens of context and you're routed to the version of the model that can handle that, the quality of the responses goes down. So we can all agree now that the version of the model that does a million tokens of context is somewhat officially according to anthropic a meaningful regression in performance. We're all aligned with that, right chat? That the 1 million token versions of the model, at least according to anthropic, are behaving worse. I'm not reaching particularly far with that, right? Well, it would really be a shame if everything suddenly routed to that dumber version, right? Like they changed in the middle of March. 1 million context is now generally available for Opus 4.6 and Sonnet 4.6. Standard pricing now applies across the full 1 million window for both models with no long context premium. I will put on the blinkers and warn you guys now. We are going into conspiracy territory. There's a few things here that have me overthinking. The first is that we now have the included full 1 million token context window. But the bigger thing I'm going to overread in the near future is the pricing. There is no multiplier. A 900k token request is built at the same per token rate as a 9K one. Previously, when you went over 200k tokens, they would charge you more per token because the routing, the management, the caching, and all of the layers to do the compute on the larger context windows was more expensive. Bigger context windows require a couple things. One of those things is more memory. Nvidia GPUs, even at the top tier, are pretty limited on how much memory they have. There are other things, like my MacBook here, that are less limited because the architecture that Apple chose for how they set this up can handle large amounts of RAM on the GPU and CPU with their unified memory standards. ARMbased chips tend to be much better at this. It is my guess that both Tranium from AWS and whatever Google is cooking with their TPUs can probably handle larger amounts of VRAM. So if Anthropic is using more compute from AWS and Google and less from Nvidia, chances are they can now eat that larger context window for cheaper. So the decision to make the full 1 million context window the default and the problem that happened with that routing before is likely that these models are some way different and possibly that these models are being served on those other places. So my conspiratorial guess here is that the Opus 1 mil context version either because of how it's served on which GPUs it's served in TPUs or just something about the model itself behaves dumber fundamentally. There's also the fact that having that much context just kind of pollutes the model. And again, it's the problem with all the pop-ups. The more [ __ ] on your screen, the harder it is to do what you're doing. And if the model isn't compacting as often, so it's constantly full of this unnecessary context, it's going to steer the direction the model goes in places you probably don't want it to go. Well, this is totally fine cuz you can just disable it, right? Let's disable it. /mod. Oh, my options are opus 47 with 1 million context, sonnet, or haiku. Huh, I can't change the context window size here. I'm just stuck. My guess is that as of the most recent Cloud Code updates, the vast vast majority of Cloud Code users are using it with 1 million context window, which as we have confirmed earlier is dumber. Anthropic has come out and said as much. The 1 million token version of the model is dumber. And here's where the deep conspiracy comes in. If Anthropic is trying to steer traffic away from their NVIDIA GPUs because they're using them for training or other things and they're steering it towards their Tranium AWS things as well as their Google TPUs, a really easy way to do that is to turn on the 1 million context for everyone because now every Cloud Code user's traffic by default isn't going to the Nvidia [ __ ] anymore. Thankfully, you can turn this off yourself by going into your settings and adding the environment variable of cloud code disable 1 million context to one. And now it won't do that anymore. If I close and rerun, now it's not using the 1 million context window. So if the 1 million context window makes it dumber, everyone is going to feel a regression because as we saw here, it's on for everyone by default and you have to go change values in your environment in order to not have that on. And I wish we could just end here, but sadly I am not the only one overthinking all of this. As I mentioned earlier, the director of AI from AMD has slammed Anthropic for cloud code getting dumber overtime. This is an issue that she wrote about how cloud code is unusable for complex engineering tasks with the February updates. They took the time to write quite the report here. Quantitative analysis of over 17,000 thinking blocks in 235,000 tool calls across 6,800 cloud code session files reveals that the roll out of thinking content redaction. The redacted thinking change correlates precisely with a measured quality regression in complex long session engineering workflows. What are they referring to here? Responses have two parts. There is the thing that you see. That could be a tool call. That could be text that you're reading. Could be a lot of different things. But there's also the thinking which is the step the model does where it kind of talks to itself to figure out what it should do. Anthropic used to just give you all of the thinking data. It just came down via the API. That was pretty cool. I liked it quite a bit. They've since stopped doing that. The reason they claim is they want to prevent distillation. And honestly, there is some truth there. If I have all of the work the model did in the API response, I can hand that to another model that I am rling and make it more likely to behave similar to how quad behaves. So they are hiding that data. What they did to keep the model from feeling really bad and dumb is they put an intermediate model that would read your thinking data and then send a summary down of it. This is the redaction being referred to here. The model not showing you all of its thinking. Instead, the model shows another model it's thinking and then you get a really short summary. The problem with this isn't just that we as users can't see what the model was thinking about. It's that the history no longer includes the thinking either. It is entirely possible for Anthropic to track all of this with like a thread ID in their database so they can recall all of the thinking data. The problem is that the API request you make can't include it anymore. When you're using cloud code and a bunch of tool calls happen, every single tool call results in the entirety of your history being sent back to Anthropic over the API. That history used to include the thinking. So all of the thinking was in that API request. That data is no longer included, which means the API request now has to not just take all the data you sent it and send that over to the model. It also has to do a database lookup to grab all of the data for your thread. So it can have that thinking data from the history assuming it does that which it seems like and I won't even say seems like which judging by the absolute [ __ ] garbage quality of engineering that tends to come out of anthropic and the absurd nature of the regressions that they've had in the past. We're talking about a company that took over a month to realize they were routing requests to the wrong model. That is insane. This is not a company that's quality of engineering can be [ __ ] trusted at all. So if we are now relying on them to do a database call to enhance the history we're sending them with the right thinking data so that it can do its job properly, it's probably not doing that. I would put it at like a 70% chance they're [ __ ] this up because they [ __ ] everything up. It's enthropic. They suck at writing code and their models are okay at it. But they are not. So assuming that my hypothesis here is correct, that the models no longer have access to the thinking data and the result is that they can just be [ __ ] stupider. It would be really bad if we were redacting too much of the thinking, right? Well, back in the day from January 30th to March 4th, all the thinking was visible. As of March 5th, it reduced 1.5% was no longer visible. And very quickly, we accelerated to the point where on March 12th, 100% of the thinking was being redacted. That data that the API used to make better responses is no longer being sent to you. So, we have to trust them to recover it themselves. And I don't think they can do that because they have never once proven they know how to write [ __ ] code. The quality regression was independently reported on March 8th, the exact date redacted thinking blocks crossed the 50% line. The rollout pattern over one week is consistent with a stage deployment. Yep. Part two. Thinking depth was declining before the redaction. The signature field of thinking blocks has a 0.971 Pearson correlation with the thinking context length. This allows estimations of thinking depth even after the redaction. They're measuring how many thinking tokens or characters are being used and the number has plummeted. It's now 73% lower than it used to be. So the models are hiding their thinking and they're thinking less. This feels very likely to be an attempt to reduce compute. If the models don't have to think as much, then they don't have to do as much and they have more compute available. This has resulted in measurable behavioral impact. Stop violations, which is how they prevent laziness, didn't happen almost ever before, and they've seen a bump from it never happening to 10 times a day. Frustration indicators and user prompts, which is the user cursing at the model saying, "What the [ __ ] What's going on?" have bumped 68% in that same window. Ownership dodging corrections needed has bumped from 6 to 13. That is more than it doubling. Prompts per session has went down a little bit, which is cool. But sessions with reasoning loops, where the reasoning gets trapped and goes forever, went from never to pretty common. This is the scariest number so far. The tool usage shift where it went from research first to edit first. I felt this one myself. felt like the model stopped looking for what to change and started just making changes. The readtoedit ratio from January to February was a 6.6, which meant that it would read six times more than it would edit. It would read 47% of the time and it would only edit around 7%. From February 13th to March 7th, it dropped to a 2.8. It got cut in half. And now it is down even further. It's now just a two. The ratio got cut to a third of where it was before. Previously, the models would read 47% of the time and only edit 7%. Now, they read 30% and edit 15%. For every two reads, there's one edit now. It used to be 6:1. That is insane. One of my favorites here is the permission seeking behavior where previously the model would just do things and now it doesn't. They had over 173 instances from March 8th to 25th of the model doing things like this is getting long or continue a new session or future work, no limitation, good stopping point, natural checkpoint. Should I continue? Want me to keep going? not caused by my changes existing issue. We've all seen these types of things recently, but this never happened before. You know, this is all for prevention of distillation. They've done a great job. Who would want to still on a model that's been this [ __ ] lobotomized? Here's a scary set of numbers here. The number of API requests has absolutely skyrocketed over 80x from January to now. There's a lot of layers to what has caused this, but even though the number of prompts has not increased from the team at AMD, they went from 7,000 down actually to around 5,600 to 5,700. The number of API requests is 80x. The number of total tokens for input has 170xed. A large portion of this is the new concurrency and multi- aent things that they introduced. The 80x increase in API requests is not purely from degradation induced thrashing. It also reflects a deliberate scaling up of concurrent agent sessions that collided with the quality regressions at the worst possible moment. The most striking row is in user prompts. We had roughly the same amount of human effort put in the same number of prompts, but the model consumed 80 times more API requests and 64 times more output tokens to reproduce demonstrabably worse results. That is insane. Weeks later, Boris responded here cites the adaptive thinking changes which should reduce this. You can disable it with a flag. Medium effort default as of March 3rd. Some people want the model to think for longer even if it takes more time in tokens. To improve intelligence, set effort to high via/effort or in your settings JSON. And as you can see from the down votes, people did not like what he had to say here very much. And if you see the up votes for the original post, you'll see many others agree. And I'm going to hit the thumbs up, too. Absolutely phenomenal reporting from Stellar over at AMD. Thank you for writing this up. This is really emphasizing my point here. The regressions aren't just in our heads anymore. They are measurable. They are demonstrable. They seem to be related to everything from bad API configurations to [ __ ] changes in cloud code itself to attempts to hide things via redaction to prevent installation and so much more. This is a disaster. Whether it's our expectations going up or a harness that feels like it was written by toddlers or an API change that makes no sense at all or GPUs that are very different that have different expectations to the model itself being served wrong or just behaving bad in general. All of these things contribute to the problem. But I do have a solution. Use literally anything else. These regressions are not things that happen with other labs. And I took a very scientific path to prove this. I asked on Twitter, "Serious question, has anyone ever noticed meaningful regressions from Codeex or OpenAI models?" They like, "We talk about this a lot with anthropic, but I've never seen a similar discussion with OpenAI." And almost nobody has said that. Occasionally for like a day generally, no. No. The only issue I had is when they release new models for a period of time, all the new models are unusable. They're slow, but that gets resolved pretty quick. Negative. I remember a bad weak week once, but generally no. And then Tibo hopping in. We've had ghosts in the Codex machines, which we publish the investigation for. This is they made changes to Codex and the Codex specific APIs, which cause problems, but we don't fiddle with the models or thinking budgets after release. We focus on keeping them up. And if we look at the measurements from the charts I was looking at earlier, you can see there is no regression in OpenAI models, especially when compared to Anthropic. This was quite a video, but I hope it helps break down the complexity of these types of things and the reality that we are experiencing where the models are getting dumber. I do not envy Anthropic and the people who have to go dig into this because there is so much surface area to cover and if I'm being frank, it all seems like it was built in a very incompetent way. The fact that Anthropic's engineering culture is [ __ ] has resulted in things being [ __ ] They need to fundamentally rethink things from the ground up if they want to build reliable infrastructure and software. Right now, we cannot trust the things Anthropic puts out. And if you feel like you've been burned, you should because you have. The $200 a month subscription you've been paying for for multiple months is getting you less and getting you worse. And that is not acceptable. I will continue calling them out until we see real change. And I hope you do, too. This is a really important thing. And I hope they wake up to the fact that they've put themselves in this corner by writing [ __ ] code, providing no transparency into how any of this works, and then expecting us to just smile and wave as the tools we rely on every day get worse. I would tell you to move to Codeex, but you guys already call me a page shell. I'm just going to end this one here. Until next time.