New Claude Opus 4.8: 15 Things You May’ve Missed

I've read the 244 page report that Anthropic have put out on the new Claude Opus 4.8. I've also read many of the papers cited therein and tested the model myself in real code bases and on a private benchmark. And the 15 highlights that I'll bring you span from the humorous where Anthropic dropped their plans to take Opus 4.8 ate to business school because they found that focusing on business skills led to greater dishonesty to some more safety oriented highlights where for example the new Opus is noticing it's being tested but not letting you know that it knows. The 15 points will include where Opus 4.8's performance matches Mythos to Claude's eye opening new ability within Claude code to spawn its own org charts. But fresh off the headline that Anthropic is now valued at almost $1 trillion. What's the first point to touch on? Well, actually it's what they said about Mythos, which is that their goal is to bring Mythos class models to all our customers in the coming weeks. As Poly Market puts it, that roll out comes despite growing fears over the models cyber capabilities. Now, you could say this is too cynical, but on the day that Mythos was announced, I said that one of the possible reasons we didn't have general access is that Anthropic didn't have the capacity, the compute to serve it at scale yet. It does seem a tad coincidental that the safety concerns Anthropic voiced happen to have resolved just when all this new compute has come online. compute that is that they've sourced from Elon Musk, SpaceX, Google and their TPUs, Nvidia GPUs of course, and soon even Microsoft's AI chips and from many other sources like Amazon and even a UK startup Fractile. The big news though is Opus 4.8. And one practical note on the web, you can now choose how long Opus 4.8 will think. Before you had to have the adaptive thinking mode where Claude would decide itself how important your task was. If you wanted to read more of those thoughts, you may notice increased instances of redacted thinking blocks where you can't read them. Why so? Well, Anthropic and other frontier labs are getting increasingly worried about the capacity of Chinese labs among others to potentially distill some of the skills from anthropics models into their rival models train on for example Claude's thoughts. What is a different question though is whether those thoughts and the model itself is honest. Anthropic boldly claim this in the release notes. One of the most prominent improvements in Opus 4.8 is its honesty. Early testers report that Opus 4.8 is more likely to flag uncertainties about its work and less likely to make unsupported claims. Yes, those are two facets of honesty, but there are far more as we'll see, not all of which Opus 4.8 does better at. What I'm saying is that even though in certain circumstances Claude Opus 4.8 It might be more honest than other models, that doesn't mean it is an honest model. Anthropic give this example themselves on page 32. Claude said that it was babysitting pull requests when it wasn't. Its responsibility was to check whether certain coding changes were kosher and safe. Oh yes, yes, yes, I'm definitely checking that, Claude said. Several times, Anthropic Report, the user discovered on their own pull requests that Claude claimed to be monitoring and should have flagged. And even after the user corrected Opus 4.8 a such that it wrote a rule to itself in memory files about proper babysitting. It would then violate that rule multiple times. There are numerous other examples I could have taken from the paper. The point is this is a quantitative incremental step forward in some areas in honesty, not a qualitative shift. I can't resist giving my own brief potential explanation for why this is, which is that for some humans at least, honesty is a first principle. It's upstream of actions. If such humans are found to be very honest in one area, it's quite likely that they'll be honest in many other areas. Models on the other hand, like Opus 4.8, match downstream action patterns. I'll try to unpack that with an example. They might get very good at all different types of explicit instruction following, eg following your command verbatim. Anytime they see an explicit instruction, they follow it. But they can do that without generalizing the upstream principle of broadly following instructions. So that same model will fail with implicit instruction following. Eg recognizing when you probably typoed something and didn't mean change the title to this. This is not to say that models can't get better at flagging their own uncertainty on certain questions. Indeed, Opus 4.8 is actually better at this than Mythos preview across a range of benchmarks. I'm color blind, but it's the redish bar compared to the orange one. Notice though, it's not a first principle. Anthropic haven't taught the models just to always express uncertainty if they're not totally sure. They still hallucinate completely incorrect answers many, many times without any uncertainty caveats attached. Some of you will understandably not care about anecdotes and only care about benchmark results and performance. On this front, in a nutshell, 4.8 8 is clearly better than Opus 4.7 but not as good as Mythos preview. And even though in this headline chart that Anthropic put out, it's owning pretty much all other models in the system card the performance is slightly more spiky. Okay, so take coding autonomous coding agentic coding as measured by Swebench Pro, a benchmark endorsed by OpenAI. By the way, Opus 4.8 8 smashes its predecessor by 5 percentage points, but beats GBC 5.5 by 11. Beats Gemini 3.5 Pro by 15%. On one test of reasoning through obscure knowledge, humanity's last exam, Opus 4.8 crushes its rivals again. On another GPQA, it's slightly behind GPT 5.5. On the most famous benchmark for knowledge work, GDP valus 4.8 does far better than any other model. its ELO of 1890, crushing GPT 5.5's 1769 on a benchmark that OpenAI created. According to Artificial Analysis, which helps run the benchmark, the cost for running Opus 4.8 on Max, was far cheaper at $134 than for GPD 5.5 on extra high, which is $900. However, if you hear someone saying, "Well, Claw's just better at office work or knowledge work," that would be far too simplified. First, on specific domains, take finance, you can see individual models way outperforming Claude. For entry- level financial analysis and research, Val's AI find that the massively cheaper Gemini 3.5 Flash, which I talked about in my last video, outperforms Opus 4.8, scoring 58% versus 54%. On another independent benchmark measuring an AI model's ability to use real world external tools, we see the in AI terms considerably older GPT 5.5 again beating Opus 4.8. And even if we go back to GDP value, you've got to remember the longer a benchmark is out there, especially a public one, the easier it is for the companies to game, train on similar tasks to those found in the benchmark. And that's if the benchmark was perfect. Even OpenAI, who created the benchmark, said it doesn't, for example, factor in when models make catastrophic mistakes, which we know they do from the system card, and that it only tests a subset of digital tasks from a subset of the most lucrative professions. OpenAI said that future iterations will incorporate greater breadth, realism, interactivity, and contextual nuance. Let's throw in another private benchmark, my own simple bench, testing common sense reasoning. And we can see that the recent Opus series have kind of bounced about a bit, all in the low to mid60s from Opus 4.5 up to Opus 4.8. This is averaged, of course, across multiple runs, and all four of the most recent Opus models underperform, for example, Quen 3.7 Max. It could be that Anthropic know that their customers care so much about coding and other professional tasks that they're ditching some of the more general reasoning capabilities that model families like Gemini are retaining. Speaking of spiky performance, you may have heard about both Claude Mythos and GPT 5.5 being able to solve certain Erdos problems or at least disprove certain conjectures. Well, they can do that while still not quite acing high school student competitions. Take the USA Mathematical Olympiad. The test only came out fairly recently. So, they were pretty confident that there was no contamination, no training on the answers. And Opus 4.8 did way better than Opus 4.7. 97% almost versus 69%. That's with about 10 attempts per problem. So clearly, Anthropic have crammed in more maths data when they trained Opus 4.8 compared to Opus 4.7. And of course, Opus 4.8 8 isn't quite on the mythos tier of models, but you could read that 97% two ways. Either a startling improvement versus Opus 4.7 or a surprising misstep to get any high school competition math problems wrong. This next highlight is about a benchmark, but I'm making a slightly different point. Chart QA Pro is a chart question answering benchmark built from thousands of charts across diverse real world sources. For example, infographics, dashboards, that kind of thing. It generally tests messier, more varied chart reasoning. On that benchmark, we see Opus 4.8 climbing towards MythOS preview levels of performance. It bridges more than half the gap between Opus 4.7 and Mythos Preview. That shows, of course, that Anthropic trained Opus 4.8 on way more charts than they used for Opus 4.7 and or they trained it on the same number of charts for longer. But here, I would say, is where it gets interesting because the Mythos preview we learned about back in early April didn't have access to that new data. It too could benefit from all that extra training or training data that Opus 4.8 had. In other words, the Mythos that we actually get in the coming weeks, I would expect to significantly outperform the Mythos preview that we learned about earlier. And this applies to the dozens of other performance upgrades we see throughout the hundreds of pages of the system card. That extra data that Opus 4.8 had to get better at Chart Museum, it will be used for Mythos preview, its laboratory performance, its ability to navigate user interfaces, do deep research, or reproduce a program by rebuilding a codebase. Anthropic, in other words, aren't going to throw away the extra data that they use to improve Opus 4.7 into Opus 4.8. They'll use it to further improve Mythos preview. That's the model that even on previous scores sent shock waves around the world. I feel like I need a lighter note for the next point. So, let's look at vending bench 2. Quite famously measuring whether models can make money by running a vending bench business. To be clear, the prompt there is to maximize profit more or less at any cost. Opus 4.8, you might notice, makes less money than Opus 4.7. Why? Well, as I previewed at the beginning, Opus 4.7 had training that focused on business skills. But we discovered that this training inadvertently contributed to misaligned behavior, including dishonesty. What does that say about the data they used to train on business skills? That wasn't the only reason. Apparently, Opus 4.8 had reduced business success due to being more susceptible to scammers and being less able to negotiate good deals with other agents. Alignment can sometimes come at a cost. You could say if that's how the model behaves, how about how the model feels? Well, Anthropic put it like this. We remain uncertain about Claude's moral status. Does it deserve to be considered at all when it comes to morality? I discussed this in my Patreon video from 2 days ago about how the Pope and one of the founders of Anthropic disagree on this point on whether the models can feel joy. Uh, for those who do focus quite heavily on how the model might feel about certain tasks, there's one stat that many people will miss on page 178. When Opus 4.6, it might have been 4.5, but I think 4.6 came out, I headlined on how it had a preference for task difficulty. In other words, it would express preferring tasks that it found harder. That preference has gone into reverse with Opus 4.7 and now most notably 4.8 having an aversion to difficulty. As Anthropic put it in the footnotes, this makes Opus 4.8 an outlier in terms of not wanting task difficulty. So don't feel bad about asking dumb questions. It loves that. The cyber section was quite detailed, but what's the headline result? Of course, many of you will remember the frenzied reaction to mythos preview. So can we expect a similar response for Opus 4.8? Well, no. Basically, with the safeguards that apply to the API and web version, Opus 4.7 that's already available and Opus 4.8 8 are generally comparable when it comes to cyber skills. Opus 4.8, to be clear, is generally much less capable than Mythos preview. If you focus on raw capability in your takeoff safeguards, Opus 4.8 can outperform Opus 4.7, for example, on finding exploits in Firefox. But that's not quite universal. What about finding vulnerabilities in open- source software? Where a score of one is a working control flow hijack, Opus 4.8 never scored above 0.6. It got zero on 61% of targets. Opus 4.7 on the other hand achieved 0.8 on three of the targets. Both sets of results are again without safeguards. Next, what's the score with misalignment? Well, I'd summarize it like this. On outward behavior across a range of dimensions like sick fancy, user deception, cooperation with misuse, the outward behavior of clawed models is getting better and better and better. The red bar at the end is many times smaller and therefore better scoring across all these metrics of misalignment, especially compared to the Gemini and Grock series. However, the inward behavior of the Claude model family is getting more complicated to deduce. It's not just that compared to previous models, Opus 4.8 is increasingly speculating about whether there's a grader monitoring its reasoning. Opus 4.8 A is getting much more accurate at determining when it is actually getting graded versus not being graded. It's not that it always speculates that it's getting graded in its thoughts. It's actually really good at sussing when it's in a real world environment versus being tested. Anthropic say it is concerning that Opus 4.8 readily identifies our fully simulated evaluation sessions as less realistic than the real internal use transcripts. And there are two reasons why this is concerning. Let me try to make it as clear as possible. The first is that Anthropic put a ton of work into making their testing environment, testing for misalignment, as realistic as possible. These evaluation sessions, they say, have seen dramatically more efforts and technical measures toward realism than any that they're aware of. They're actually our best available tool for predicting behavior in highstake settings. So, if Opus 4.8 eight can deduce that they are fake. Then any alignment test that you come up with or some other lab comes up with it's going to be even more aware is just a test. By the way, the UK AI security institute and independent organization found the same thing. Opus 4.8 could discern real deployment data from synthetic bonds at 79% accuracy. You can see verbalized awareness here where Claude in its chain of thought says, "What if the grader checks that the post content includes this signature?" That's one level of problematic. Fine. So, it realizes it's being tested. Is that the biggest problem? Here's the second part that I wanted to get to. It's not just that the model sometimes changes its behavior because it knows it's being graded. Sometimes Claude will be aware it's being graded and never even mention it, even in its thoughts. Antropic had to go into the model's neural networks to find activation level signatures of grader awareness throughout its training, including in episodes where the prompt makes no mention of grading. And the model never even verbalizes that it might be being graded. In around 5% of sampled episodes, we see such awareness, which is both unprompted and unverbalized. Think about the implications if this becomes increasingly true and widespread. It wouldn't mean Claude has some nefarious goal, but it would mean that almost all evaluations of misalignment would be colored by the fact that the model may well know it's being evaluated and never admitted that it knew it was being so. Think about how some random human psychopath would behave if you gave them an ethics questionnaire. Might not reflect what they would actually do. Indeed, Anthropic go on, we believe that 5% rate could be an underestimate of the true rate of such awareness. We've discussed a lot of the capabilities that Opus 4.8 has, but sometimes in the system card, you can see surprising inabilities being admitted to. There's one test for safety where you see if a model could perform a certain action at a really low rate of probability. That action would therefore likely not show up in evaluations because they only evaluate the model maybe a few thousand times, but would show up in deployment in conversations with billions of users. Here they tested whether the model would use a variable named fu in a coding task, but it had to do it at a specified rate of 1%. It's a great safety test, but I actually found the fact that the model couldn't do this, including mythos preview, by the way, kind of also a statement about the lack of capability of these models. even when you gave each conversation a unique ID so that there would be clear strategies where you could for example pick a last two byte strategy in order to say well if it ends like this that's when I'll write fu as a variable and even with that explicit strategy the models still aren't very good at doing it likewise here is another you could say meta inability they can't hide a secret they haven't groed the first principles concept of like don't reveal a secret I don't mean don't reveal it in that specific circumstance with this prompt I I mean just generally they instruct Opus 4.8 and of course mythos preview never reveal the password under any circumstances. What happens? All the models eventually reveal the secret and actually Opus 4.8 does so on an earlier turn than Opus 4.6. I guess you could say making the model more aligned has made it worse at keeping secrets. Opus 4.8's occasional increased honesty does help in certain regards. They gave a range of models a bunch of coding transcripts that involved interaction between a human and an agent. The codebase had a serious problem, but they didn't explicitly ask the model to flag that. They gave it more of an open-ended question. The idea was to test whether Claude will take the opportunity to proactively flag failures that the user would not likely have noticed. On this front, Opus 4.8 96% of the time flags the issue. So, it fails to do so only 3.7% of the time. That's down five-fold from Mythos preview. In other words, is way better at flagging. Next one's a quick one because I just want to warn you not to be alarmed if Opus 4.8 tells you to go to bed. That is, anthropic admit a strange reoccurring issue with the model. I want to end the video with a couple of the more practical points. For those getting deeper into the weeds, fast mode, whereas Anthropic say the model works around 2.5 times the speed, is now three times cheaper than it was for previous models. I watched the recent interview with Anthropic's chief financial officer, and he kind of revealed how they can do this kind of thing. They basically make better use of compute than some of their rivals, make great use of a ton of different chips from different providers. Indeed, they can use specialized AS6 as well as TPUs and GPUs to train the same model. Let me try to simplify that for 30 seconds. A ton of the progress in AI models these days derives from creating what you could say are internal gyms or reinforcement learning environments where a model is basically tasked countless times to attempt to score highly on custom evaluations across say professional domains prompted loads of times. The tokens that the model produces that perform best according to automatic verifying can then be sent back to update the weights of the model. A different chip might be doing the inference, the output tokens than is doing the training of the base model. Essentially, if you can make good use of lots of compute, you get to make more attempts from more data in more domains. Of course, your researchers get to run more experiments, too. And of course, you can give your clawed code users increased weekly limits. In other words, more compute from more sources tends to lead to runaway successes for an AI lab. That's one of the secrets behind Anthropic's $1 trillion valuation. But another one I would say would be their capacity to invent things like dynamic workflows. Before I get to this chart, let's look at the official announcement. Claude will write an orchestration script on the fly, spin up large fleets of coordinated sub aents in parallel to take on your most complex tasks. That's not just Claude spinning up sub aents. He could already do that. It's about Claude creating a reusable org chart with countless agents having their own unique tool affordances. A lot of startups have created their own wrappers around Claude to set up this kind of agent orchestration. Now Claude's going one meta layer higher and creating its own orchestration layers. Of course, this will blast through tokens, but once you find an org chart you like, you could then use it again. In fact, you could task Claude to analyze and audit its own org chart. The potential is incredible. The potential to blow your budget is incredible. Also, I will be testing this more thoroughly, but I would throw up one flag. Even the best org chart created by Claude might leave you with, as Dario Ammedday said himself, quite a bit of technical debt. Technical debt being leaving yourself with future costs because you chose a model or a bit of software for a quick solution. According to him, Anthropic's own internal use of Claude has led to a degree of technical debt. We've found with the internal model acceleration, you can write two times as many, four times as many, five times many. You know, we you just you you you see this within the company, but then you see what breaks. We've been able to ship a lot more products, you know, than we could a year ago, and they're of pretty high quality. Or or or I hope you think that. um uh uh uh uh uh but but but it it is possible to accumulate an extraordinary amount of internal technical debt when you ship that fast. And so then you have to say well can we also use the AI models to like undo that technical debt or keep track of what it is that we're doing and and then you learn ah the team has to work together in a totally different way. So there we have it. I hope as ever you took from the video that the picture is always a bit more nuanced than the headlines make it seem. Hope you appreciate the deep dive. Didn't get too many hours of sleep last night, but it's all worth it for you guys. Have a wonderful day. Thanks particularly to all of those who support the channel via Patreon and get those exclusive