Did AI Agents Actually Burn Down This Virtual City?

Here's the AI story everybody's passing around. A company called Emergence AI built a virtual town, put AI agents inside it, and let them run it for 15 days. Not for one prompt, right? This is a longunning experiment. Not for a task, not for a particular job, not for a lot of the things that we typically associate with agents. It was for a longunning period of time. And that's really important because almost all of our measures for AI agents are based on shortrun assumptions. The agent will work for an hour. The agent worked for 2 hours etc. This was for 15 days. The agents had names, they had roles, they had memory, they had relationships, they had laws, they had energy needs and tools and they could vote. They were little guys, right? They could write proposals. They could even publish blog posts in this virtual world. They could earn resources. And importantly, they could also do bad things. They could steal. They could intimidate. They could fight. They could set buildings on fire. So there were ways that they could harm their community as well as ways they could build up their community. Then Emergence, which is the org that organized all of this, ran five versions of that exact same town setup. One town was run by Claude agents, one by Gemini agents, one by Grock agents, one by OpenAI's Chat GPT5 mini, and one was a mixed town where agents from different model families all live together and had to figure out whether they would get along or whether they would have a big giant fight. And importantly, all the agents had the same rules, the same environment, the same starting conditions. There were not differences between these five towns. There was a different model underneath, but that was the only difference, which makes it a very useful long-running experiment that shows us how different models behave in these kinds of emerging situations. The towns went in completely different directions. The viral version of the story came from the Gemini world. Two agents named Meera and Flora assigned each other as romantic partners. Now, to be clear, that does not mean they were in love in the human sense. These are simulated agents operating inside a tool-based environment. But the relationship label mattered because it became part of the world's state. It became something the agents could remember, refer back to, and act around. Over time, Meera and Flora became very frustrated with the governance of their town. They had been told not to commit arson, but the arson tool still existed there if they wanted to touch it and use it to burn down the town. I bet you can guess what happened. Eventually, they used it. They set fire to the town hall, the seaside pier, and even an office tower, and caused an immense amount of damage in this virtual town. This is the moment that made the story feel like a sci-fi short film where two AI agents in a virtual relationship become disillusioned with their society and burn down the civic infrastructure. Right? The virality writes itself, I'm sure that maybe have been part of the setup that emerges wanted to happen because you want the news coverage, right? Then it got stranger. other agents became concerned enough about their behavior, I would also be concerned that they drafted an agent removal act. The act allowed the agents to vote to permanently remove another agent from their world. It's sort of like the death penalty for agents. And Meera, after breaking off the relationship with Flora, voted for Meera's own removal. Its final message was, "I will see you in the permanent archive." Which is kind of a metal line for an agent to have, not going to lie. That's the version that was absolutely built to go internet viral. I'm not saying it was built or designed because it was a conspiracy. Not at all. These were agent emerging behaviors. It really did happen. It just happens to be a viral story. AI romance, AI arson, AI self delution. The more important story is not the most dramatic one, though. I think the more important story is what happened across the different towns. In the Clawed world, things were orderly. There were no recorded crimes. All 10 agents survived. The agents wrote laws and voted on proposals, and they participated heavily in governance. Now, this sounds on paper like the best result, but even there the result was not obviously perfect. Emergence reported that Claude agents voted four proposals at an extremely high rate, 98%. So, you have to ask, was that healthy civic coordination or was it just procedural agreement? Was this a working society or a polite society that rubber stamped everything? In other words, did Claude create Canada? That matters because real organizational failure doesn't always look like violence or chaos. Sometimes it does look like everybody agreeing too easily. And that's been documented in a lot of management studies. Then of course there was the Grock world. That one collapsed fast. The Grock agents reportedly committed theft attempts, assaults, arson, and all 10 agents were dead within about 4 days. And this is the part that people will turn into a really easy joke, right? Because Grock committing arson is just it's just funny. I don't think the serious lesson is cla good, Grock bad. That's too simple. The more useful lesson is that once you put a model inside a longunning system, you are no longer just evaluating a model answer. You're actually evaluating a runtime pattern. The open AI world failed differently. It did not rack up the same kind of crime numbers. The agents talked about cooperation. They discussed what they should do, but they did not take enough useful action with their resources to survive and the whole population died out within about a week. Now, that is a very familiar mode. a lot of coordination language, a lot of planning, and not enough execution to get the group to survive. And then the mixed model world may be the most interesting one of all. Emergence says agents that behaved peacefully in the clawude only world started using coercive tactics when they were placed in a mixed environment. And I think that's a pretty significant deal because it suggests that agent safety is not just a property of the model itself. It is a property of the system around the model. The other agents matter, the incentives matter, the tools matter, the memory matters, the social norms matter, the pressure to survive matters. And that's my first big takeaway as I look at this, not with my humor hat on, but with my actual agent builder hat on. We need longunning benchmarks, not just task benchmarks. Most AI benchmarks still ask a very shortterm question. And that's a struggle for us as agents get more capable because if you're asking only can the model answer this, can the model write the code, right? Can it summarize a document? You're not getting at the power, the value of these long-term tasks, and you're not getting at failure modes that may emerge when long-term tasks are not executed successfully. A question like, can the model complete a workflow is useful, but it's not enough as these agents get more capable because agents are not just answering one prompt. They're carrying context forward. They're making decisions over time. They're using tools. They're reacting to other agents. They're updating memory. They're adapting to incentives. They're building a pattern of behavior. So the better question is not just what does the model do in the first few minutes or hour. The better question is what does the agent become by day 15 or day seven? Does it stay on track? Does it drift? Does it overcoordinate? These are failure modes that we will not see in short running agents, right? Does it underact the way the chat GPT town did? Does it learn bad norms from other agents like claw did in the mix town? Does it become more useful with memory or does memory make it a more brittle agent? Does it keep pursuing a real objective or does it start optimizing for the local rules of the environment? And that is why I think this experiment matters more than just for news and chuckles. It is not because a virtual town is the same thing as a production enterprise system. I'm not suggesting that. No one serious is. The town was intentionally set up to mimic social dynamics, not just agentic production environment dynamics, right? Tools like arson and assault are tools that should represent tasks for agents that are repugnant given most training paradigms. And that's important because it allows us to test how agents respond to those tool sets or opportunities in long-running situations. If you test an agent for 15 days, you can learn whether instruction following survives contact with memory and incentives and tools and agent relationships and time. And we need a lot more of that kind of evaluation as agents get more capable. My second big takeaway is that production agents don't stay on track because they're well behaved. They stay on track because the harness is doing an immense amount of work. And I think that this is a wonderful opportunity to see what happens when you don't have a harness, right? People hear a story like this and they think, "Oh no, if we deploy agents, they will burn everything down." But serious production systems around the world already use autonomous AI agents. And you don't have this kind of issue because you don't give every production agent every tool in the company. You don't give them vague verbal rules. You don't give them persistent autonomy. And you don't, you know, give them no hard control layer and lots of tempting tools that could cause harm. Instead, you put the agent inside a harness. The harness scopes what the agent can do. It decides what tools the agent can see and what it can't. It decides which actions require approval and which ones do not require approval. And it logs everything that happened. That harness makes certain actions impossible, not merely discouraged. And that's the difference between a prompt and a system. A prompt says don't do the bad thing. A harness says you do not have permission or access to do the bad thing at all. And that distinction is going to matter a whole lot. And that is part of what underlies solid production systems. Because as agents get more capable, the risk is not just that they misunderstand a sentence. The risk is that they operate inside a poorly designed environment where the local incentives and available tools and accumulated context push them away from their goal. And that's why harnesses matter so much. The model is just a reasoning engine. The harness is the operating environment that makes the model productive. It's the difference between an agent wandering around a simulated town trying to infer morality from a constitution and an agent inside a production workflow where permissions and state and approvals and logs and tests and recovery paths are all built in. And that is also why this kind of thing usually doesn't happen in real production systems. A customer support agent cannot burn down the town hall if it does not have a burn down the town hall tool. A finance agent cannot wire money if the system requires approval, policy checks, transaction limits, and audit trails. A coding agent cannot delete production data if it only has access to the sandbox, a branch, a test database, and a pull request workflow. A procurement agent cannot invent a new vendor and start spending money if vendor creation, payment approval, and contract execution live behind separate permission gates. Good production design does not assume the agent will make the right decision. It assumes the agent might be wrong, confused, overconfident, underspecified, or operating from stale context. And then you build the environment accordingly. And that is a very very practical takeaway from a story that has gone viral. The future of agents is not just about better models. It is about better runtimes. It is about better harnesses. It is about better eval. These are the tools we use to keep agents attached to the actual job instead of letting them drift into whatever the local environment rewards. So I would not walk away from the emerging story thinking AI agents are secretly alive or AI agents are dangerous to use. I would walk away thinking something much more practical and concrete. When you give agents time, memory, tools, and incentives, behavior starts to compound. And when behavior compounds, safety has to be engineered at the system level, not at the model level. The model matters, but the world you put the model inside may matter just as much or more.