Unlocking agentic AI: Secure Sandboxes are officially GA

Hey Kate, what's up? Good to see you. Hey Mike. All good. How are you? Really nice to chat to you. Yeah, I'm excited to to talk to everybody right now. So, we are launching uh sandboxes in GA this week. We've got a blog post out about it which you should go read right now. Um and we're going to talk about it. And then also we want to talk about that and Gentic O and sandboxes which we think we have some really cool patterns uh that solve a lot of problems for how to you know stick something like clawed code or open code in a box and do really great O to keep everything secure. So Kate and I wanted to talk to each other and talk to you all about that today and give you an overview of uh what we've been doing. We should introduce ourselves though. Kate, you want to go first? Yeah. Uh totally. Uh it's a good idea. So hey everyone, I'm an engineering manager on the agents and sandboxes team and I lead the team that is implementing the couple of things that we're going to be discussing today. Mike, what about you? Yeah, uh product manager at Cloudflare for about two years. So work on the containers team which our sandbox SDK is built on top of as well as some other teams like the agents team here at Cloudflare. So excited to be chatting. So let's start with uh G really quick. So that's kind of the the big announcement right now and and the thing that's prompting a lot of this. Containers and sandboxes are now going G, which means that, you know, we're fully supporting them. If you're a big customer and you want to say, "Hey, let's bet on this." We're saying that, you know, Cloudflare is is ready to back that. And there have been a lot of changes that have we got them out in beta last June. Been a ton of changes since then. So we wanted to take this opportunity to go over some of the changes. I'd say some of the biggest ones are just to call out first are are limits. So right now you can have if you're just like a a standard user swiping your credit card for the first time up to 15,000 of our smallest container or smallest sandbox running in prod larger sandboxes thousands of those if you want talk to us if you want even more. We've got people running you know tens and thousands of concurrent sandboxes at scale. So that's something we're really excited about is is that's been a really big change. And then just to call out another pricing change that that came out recently where we charge for active CPU now. So, these have been two really big changes, but there have al also been a ton of usability changes to to sandboxes and containers that we'll go over soon. So, Kate, I mean, what should we talk about first? Uh, well, I think we should start with um some core premises. Essentially, a lot of people right now, including us, are saying that, hey, AI agents need something more than the CLI. they need a real computer to to to do many tasks and you know spawn sub agents write code etc. Uh I'm really curious what do you think what were the biggest pain points you saw companies hitting when they try to stitch together agent environments using traditional VMs or existing container solution. Yeah. So I I feel like that that comes down to like what what is a sandbox, right? And and that's something that like there are a bunch of actual answers to and I feel like there's the core idea is that there's this like trusted and untrusted areas and there's the untrusted area which usually you stick some sort of like agent to harness LLM driven agent within that untrusted area. This could also be used for just like generic user code that's not necessarily even agent driven but it's obviously you know becoming a big thing because of agents. So you have that kind of untrusted area and then you have some like you know isolation software level or hardware level around that right and then you have this kind of like trusted level of control and so you as the platform running agents or or whatnot can say hey I'm going to spin up a sandbox I'm going to you know route traffic to it in this way I'm going to control the traffic that comes out of it in certain ways I'm going to control the file system in certain ways so you have this like layer of trust that's controlled around the sandbox and then you have whatever's happening in the sandbox which you know hopefully it's it's not nefarious or anything bad but you kind of have to assume that something bad could happen within that sandbox and when we were talking to various companies basically like the first thing people started with were all these container solutions right that that were kind of like offtheshelf hey I'm going to like use my Kubernetes cluster and I'm going to run a bunch of containers and run agents within them um and and this had a bunch of issues like some of which were were ergonomic issues of just like how do I spin things up quickly because a lot of these like I expect there to be something on demand that has a file system in a certain state. That was an issue. Another issue is just like how do I route? Like a lot of these solutions were built for these kind of like horizontally scalable apps all of which were like interchangeable and that's just not the case anymore. Like the kind of new world is a bunch of individualized stateful instances and you have to route to a specific one. So like those were two big ergonomics issues. And then there is this issue of just like hey how do I do the security layer? So like if you have a bunch of containers that are sharing the same kernel on the same machine that opens you up to a bunch of potential attacks, right? Like you could have like hey I I compromise this machine with my agent and it does a spectre attack on on a you know a container that's running right next to me. And like that's really scary. And so there's this security layer around sandboxes that's that's really hard for people to solve unless they've they've spent a lot of time thinking about that. So there's the kind of ergonomics layers and then there's the security layer of like hey how do I actually isolate these things from one another in a way that doesn't you know involve a PhD and and a team of you know 10 SRDs figuring this out. You know, Mike, we we hear the word sandbox quite often, and I was wondering if it would be a good idea to highlight and sort of um provide people an answer of when to use sandbox as a like full-blown computer or is there any other ways to use similar primitives for a bit more lightweight tasks? What do you think about it? Yeah. So, we actually uh have two things that are kind of in the sandbox realm at Cloudflare. And one of which is a full kind of microVM that's that's kind of like, hey, if if you have an agent that is embodying a developer for a little bit, right? Like, hey, you're doing code review, you're checking out a multiple git repos and in different languages and having to do X, Y, and Z and run CLI tools and whatnot. We say, hey, you should use something that looks like a full computer, right? Linux box, right? that that has that layer of isolation and control that I talked about. But then there's, you know, so there's that the kind of agent as developer primitive or pattern and we say, hey, go use a full Linux machine. Then there's things like, hey, if you imagine like I'm running chat GPT or claw and it runs like a little snippet of Python for me, right? Like you don't need a full or like a little snippet of JavaScript here and there. Like that's still an agent that you want to sandbox and do something in a secure way. don't want to spin up an entire container, an entire VM that's that's potentially lightweight. You don't want to pay for it. You don't want to wait for it just to like run a little snippet of JavaScript or like taking it one step further like you want to kind of host an application that has been vibe coded long term. You don't want to necessarily run like an entire sandbox container for each of these applications. You still want this layer of control. You still want this layer of like trust like this trusted boundary. But what we have is something called dynamic isolates that we're really excited about for for these use cases where if it's just like hey I want to run a little bit of JavaScript or a little bit of Python or something that compiles to web assembly in a way that's that's faster that's that's cheaper that's easier to distribute globally. We say hey let's use a dynamic isolate for that that still has the same kind of pattern of there's this layer of trusted control around the sandbox but then there's this kind of untrusted center that can execute code. Um, so we kind of have both of these patterns which is is kind of sandboxes TM which is hey if you want to be a full developer go use that but then if you want something that's like lightweight and and just runs a little bit of code we also have dynamic workers and dynamic isolates for that. Um, so I think we're going to talk mostly about the the first one, right, which is like trying to use a full computer. That's what's going G today, but definitely check out some of the other Cloudflood blogs that have come out as part of agents week if you want to if you want something like again super lightweight that you can run some code in. Um, so Kate, I'm I'm going to flip it back at you now. So Kate, you've been running the the kind of sandbox SDK team and a lot of what you do is like a kind of ergonomics around like making these things really easy to use at scale when you're trying to like embody a full developer. You're trying to give an agent a full computer. Um can you talk about some of the problems that people have faced and like how we're solving them? How we're making it like easy to do X Y and Z? Yeah, absolutely. I think we we sort of want to replicate how a real developer would work with their computer and how their development flow would look like. So essentially you want this secure isolated environment which is by the way technically powered by our Cloudflare containers. Um this environment would be able to spin up on demand. It can sleep when it's idle. Um, it can preserve its full disc and state using the R2 snapshot that we recently released. So, whenever the developer want to come back to a task that they like, let's say postpone while they're getting lunch, they immediately get their environment back and it's ready to go. Obviously, there's a couple of very useful tools that developers use is like the terminal. You always use a bunch of CLI. So you do want your computer to be able to use a bunch of terminals, have a bunch of tabs and being a able to open it. As a developers, we usually sort of rebuild project, rebuilds UIs, front ends, and we want to have feedback immediately and see the changes immediately for for this sandbox comes with the file system watching which lets you immediately reload and see the changes that you've just applied um through your AI agent or however you want. The biggest thing also comes with the secure egress networking because everything you do you want to do with security in mind. You you don't want to let your agent go and just go completely like wild and and blast things and disclose your data, disclose your keys, you know. you you want this sort of out of the box solution where you can inject your secrets and sleep calmly during the night and not be worried about losing them or someone stealing them. So I think that's what we're trying to solve. Essentially all these um basic things that every developer works and sort of uses battles with and trying to solve on a daily basis uh comes from um cloud fire sandboxes out of the box and you know we try to solve most of the pain points that people experience. Yeah. Yeah. saying I was going to I was going to say like there's like a pattern I've noticed for like either CI jobs or vibe coding platforms or things that are adjacent to vibe coding platforms where it like the user has some git repo they they want to like do development on it right agentic development in some way and then they may or may not like start a session and then they may or may not come back to it later and can you just like walk through like what that looks like on sandboxes like from like user logs in and says point to this g get git repo to like code a shipped like what would that look like in terms of how you code that? Yeah, for sure. And there there's a bunch of flows you can use. Obviously, a lot of people right now are using tools like Claude or Open Code or other coding agents. You absolutely can run your uh coding agents inside the sandbox and call the coding agent and make your coding agent start a project for you. You can also use just um um you know old school traditional shell and say hey get clone and clone something inside the storage of a sandbox and I think from there as you work with any project you anything that comes with let's say npm the first thing would be to do npm install so you get a bunch of nod modules you have a bunch of baggage that basically comes the moment you start working on your project and you don't want to lose the progress like let's say if I'm working on something and I want to kind of freeze this state where I am right now and come back when I have a bit more time or I've come up with idea how to architect this better like how do I make sure that the moment I have this I know how to do it I immediately come back go can go back to exact point where I stop without waiting for my machine to prepare a green cell reexpose things like this is a problem and you know humans they're not very patient ient creatures they want things like instantly immediately and I think we want to account for that. So what we've done with the sandboxes is we tried to just minimize this time from you abandoned your coding agent, you abandoned your sandbox to it's immediately available to you within seconds. That's where the backups from R2 comes. What can happen and what you can happen using our backups. It's just the whole state, the whole disk, the whole memory of your sandbox can be uploaded to our R2 storage and that includes your installed node modules. So all the changes are going to go exactly how you left them in the R2. And instead of waiting about like 30 seconds when you come back to to kind of reinstantiate your sandbox, install everything again. So fully prep your like dev environment, you just can't get it all back immediately in the same state within two seconds. And I think this is what people want. They want this like, hey, we're switching context. We're going from this task to this task. And you don't have to wait seconds or minutes between the it depends also on the complexity of the project. If the project is a gigantic monolith application it you know the installation setup of the basic work environment can take quite a few minutes. We want to minimize that. We want to bring this like magical experience start where you left off. Highly recommend for everyone who is interesting to just speed up your sandboxes and get to much faster cold starts. take a look at our backup API. We've really tried to um keep a lot of DX um wishes and feedback from users while we're designing this API and yeah, I think it works pretty magically to me. Yeah, it's a hard problem like especially if again if you're like taking an existing container solution like if you're just like used to running web apps like how often is it that you need to like very quickly sync state somewhere like like a lot of these sandboxing tools including ours right like that's that's one of the kind of core things you have to get good at is like hey we have some state that's like not being used right now we need to like take that those files that that disc that npm install and just like shove it into a sandbox you know super fast such that your end user doesn't care Right. And and doesn't like you don't have to like take that hit again of of doing the mpm install. So that's that's been a super common pattern I've seen is like spin up individualized sandbox get clone npm install take a snapshot and then come back to it later if you want. Cool. Exactly. Um what else should we talk about? Um I think we should touch upon the well I I said there are two different flows. One is you get access to things through your coding agent whatever you run in sandbox and the other one is like you can run the CLI yourself and I think one thing we give users out of the box is the pty it's something we've released I think in February this year and it's basically a pseuda terminal session in the sandbox that gets proxyed over the websocket connection. It's also fully compatible with exterm.gs and a bunch of other add-ons. It it's not limited to just externs. You can extend it as as you wish. Basically the idea is basically that agent needs shell access and it shouldn't be just like run command wait run command wait then check the transcript. It should be naturally. It should be exactly how developer would would use it. Like humans run something, they check the output, then they may be interrupt interrupted, then they maybe completely forget about it, they go away, but they want to come back and then continue using their CLI, right? So we kind of wanted to give the same feedback loop. And essentially in the PTY, each terminal session gets its own like fully isolated shell. So it has its own working directory. its own environment etc. You can open as many as you need just like you would do on your personal computer during development and yeah you can reconnect and see the replay of everything you missed. So essentially this is a full-blown working environment as you would have on your laptop. Yeah, I I love that especially from debugging. We've got uh you know we're running agents internally and like the the idea just like hey I could open up a web page and with like just a tiny bit of code that you sprinkled in basically have shell access behind our off that we've set up right on our our you know UI that we provide all of our developers. It's super powerful to have that. Uh, and then of course you could give that to an agent too, but it it's useful for both, you know, agents and humans to be able to just like shell into something and you don't have to do some like, you know, horrible setup to to make it all work off-wise. It just like kind of built into your flow. Cool. I kind of want to talk about the the egress topic uh a little bit because I think there's some exciting stuff there. Absolutely, Mike. you know many times as a when I was young and junior I exposed secrets as all of us did you know I don't think there is a perfect developer who's never done that and I think we we want to make sure and help people to avoid these cases going forward and and you know build a solution that helps to prevent those things um I know you guys worked on egress for sandboxes that makes this possible and makes it really nice experience Do you want to walk us through how this feature is designed and how it works and what kind of experience the end user would get? Yeah. Yeah. Um, so so like let me frame the problem I guess like say the problem is you have some internal data that you want to talk to. So like say this is like I it could be a CRM it could be like your internal GitHub or your GitLab instance or whatnot that you're running that with code or or or whatnot. Um, so internal service you want to talk to and you want your agents to be able to talk to that. And there are a couple ways that people kind of do this off. Like the naive way is like mint a long live token, stick it in the box, and then like every agent has access to it and you don't know how they're going to exfiltrate it. So like that's really bad for like hopefully obvious reasons, right? They you know the token's out there forever. Other times you'll do like shortlived tokens and this like helps the problem a little bit, but it can still be leaked and like you could still get your data like you're still giving a token to like an potentially untrusted source at that point, which is also bad. Often people are doing like OIDC tokens for this and and this is like hey we've encoded the identity of the agent in a token and we could do some like exchange out of band to say like hey I present my OIDC token it checks against some policy and then that is exchanged for a shortlived token that has access to X Y and Z and like that's pretty good a lot of people are doing OIDC exchange but it's kind of hard like you need to set up this like whole token exchange system and you sometimes need to rely on your third parties to have this set up and have good arbback and whatnot. So we thought like hey we want a system where it's really easily programmable like I can set any policy I want around my sandbox. So I can say, hey, even though my third party provider doesn't even have the the like token capabilities, I want to lock down to these specific API endpoints, I never want to hit have a body that has this content within it and could be checked. And it doesn't matter if the third party provider supports X, Y, or Z. Like I should be able to program that as the sandbox kind of owner what it can or cannot do. So we want it to be super programmable, but we want it to be fast where you're not having to like go to some centralized source for O details and like check like it should be something that's just like ideally right next to the sandbox and we wanted to be kind of identity aware so like different sandboxes could have different policy so I could say like this sandbox has access to my my internal service fu this service doesn't have access and like maybe you could even turn the access on and off as it goes so there are kind of all these like things we wanted to do and and of course like no secret ever exposed to the sandbox basically it should only be injected on the way out as the traffic leaves the sandbox and So what we've done is we've we've we've used something called outbound workers to achieve this where essentially you can opt in on your sandbox or your container to say hey at this host name like if I am hitting you know my version control system.com let's say we are going to intercept that traffic we're going to route it wherever you want and then we are going to be inject credentials if you want and we could even like modify anything about the the request. So the kind of the most common thing is hey request goes to this destination it checks your identity against some source it also checks some policy and then it injects that credential on the way out and and this is I think way more powerful than a lot of what other sandbox providers are doing like the kind of basics is like a lot of people are like allow list denyless right where it's like you're allowed to hit mpm and GitHub and nothing else and like that's okay but like you still have this off creds problem and so we're really excited about. You have this allow list, deny list ability, but you also have the ability to inject creds on the way out. Again, the sandbox never sees it. It's just part of the request. And then um also it's totally programmable. So you could turn these policies on and off on the fly. And you can do things like, hey, only allow these endpoints or only allow, you know, post request or only allow get requests, don't allow post requests. Like it's just like super trivial to to program these things, which is is really nice. It sounds super interesting and it seems like users can do quite a lot of stuff with it and customize it and sort of you know do a bunch of things. Where do you think people can learn more about how to do sandbox off? Can you share more? Yeah. Yeah. Uh so there's a blog post that that just went out. Um so if you check our blog post I think it's called dynamic uh off dynamic identityware aware secure sandbox off. So check that out. Um again it's it's super powerful. It can be as simple as allow list denial list if you want or you just pass a function into your sandbox that says hey route all traffic through this and then the world is your oyster as far as what you control. Yeah. So, so we're really excited about that and we think that that's kind of like where sandboxes are going in the future. Like everybody's going to have this programmatic proxy that sits right there that's trusted. Make sure that that nothing is ever exposed to the agent. Cool. anything else that I think there's maybe one more thing we should mention and highlight we are talking again about the natural development cycle manually or with a coding agent you clone the project you make modifications you make some changes uh we already said that you can like hot reload in real time and see the changes but what if like I've done some cool stuff and I want to share it with Mike like how I'm going to do it you know I'm going send him saying local host that would be wildly inappropriate. Um but what we can do in one how sandbox SDK solve this out of the box is we have something called preview urls which is something that deployed on cloudflare infrastructure and you can immediately publish the output of your coding agent and show it to other people by like letting them use that preview URL. I think it's super important because like now with agents sub agents and a bunch of coding just assistants in our work cycle we want to share those results with our team you know the cl like cloudfire sandbox makes it very easy how you can do that you can start a process you can expose ports you can you can always do um and share live directly from your sandbox you don't have to even exit it um we have additional really cool utilities where you can just make sure everything works as expected act it. You can catch specific logs. You can expose specific ports. You can do all the real work that you would usually do instead of just like, you know, screenshotting your results and sending people through Slack or Google Chat, etc. Um, yeah, I think I love that for verification, too, because that that's feels like such a big part of making like a useful agentic flow, right? Is actually verifying your results as you go. Like sometimes, I don't know if you've had this experience, you set up your agent and it doesn't have a good way to verify what it's done. And so it just like oneshots code and then you have to go like spin the server up yourself and then be like, "Oh, agent, you did this wrong." Like do blah blah blah. And like ideally, you set up your agent where it's just like, "Hey, check your results against this live running server." And it makes it really easy to get that kind of iteration flow, whether it's you and an agent or you and another person, right? Like sharing this back and you don't want to separate those two things. you want to have all in one place and let your agent have all the context they can get and you know edit things in the real time. So no, it's absolutely amazing. I think you know as I'm guilty, I'm sure you're guilty. I love to have a million different instances of coding agents that are all separate sandboxes and I'm pretty sure I myself hit our internal limits quite I mean quite a lot I would say every day and I think a lot of our users are very creative and they build projects where you know usage of sandboxes is high and most of the times even critical. Um I would love to ask you more and if you can share what are the limits that we are exposing to the users um with the GA um and being public and what kind of instances the users can get um from from our solutions. Yes. So uh let let me pull up the the official limits right now. One second. So we have our smallest size which is pretty small. Let me get the the right numbers so I don't accidentally say something incorrect. So essentially you're able to do like I think up to 15,000 of our light instance but that's like a really small instance right so that's you know 16th of a CPU 2 gigabytes of disk and you know under a gig of memory right and so so that's you know 15,000 of those that's really nice for again those like super lightweight code execution that doesn't need to be like run you know if you're you don't want to like run the JVM in that or anything right once you get up to kind of the standard size we mostly see people v coding with about like a full vCPU U somewhere between 4 to 6 gigs and then around 10 GB sorry gigabytes of memory and then around 12 GB of disc or something that you could run over a thousand concurrently of that size and then you could also go up at larger sizes as well. So you know you could get up to four vCPU 20 GB of disc and and 12 GB of memory. We can also go higher than that. This is something that as you know we're GA now but that doesn't mean that we're stopping pushing these limits higher and higher. So we do have have uh you know people who in terms of total concurrency we're working with them to run they have run 40,000 of uh of these kind of like midsize instances in prod have run 80,000 concurrently in tests. So we could get really really high if you're like supporting a massive amount of users running agents on your platform. Talk to us if that's the case. Publish limits right now are either tens of tens of thousands of the smaller instances or or a couple thousand of the kind of midsize instances. Yeah, I mean as a team and as a product we want to help you scale and I think we kind of have everything we need on our side to make you successful and let you run it with as many users as the product requires. I think it's exciting to know that there have been some pricing changes to the containers product. Do you want to tell us more about that? Yeah, the big change was the the active CPU pricing. So when you run one of these just kind of the the baseline of how these things are charged is right you call start on a container you're charged from when that container starts booting to when it ends. So if you have a threeinut coding session you're charged for those three minutes. You can also set a sleep after which is really nice to say hey essentially if I haven't gotten any commands to like execute code or requests to you know HTTP requests the container will automatically sleep. And so you're generally charged from from that time either you know you start to the time you tell it to stop hey stop this container or to the time where it automatically kind of times out. Now during that you're charged for disk and memory. The big change recently was you're only charged for active CPU cycles during that time. And so if your CPU usually again going back to that npm analogy, it might be, hey, I I boot my my process, there's a little spike in CPU to, you know, boot up my web server and then there's this like big spike as it npm installs and then as I'm like waiting for the LLM to respond to me or I'm waiting for user input, it's like sitting at 10% CPU utilization. And essentially whereas before you used to be charged for kind of like your high water mark of like all the CPU you used during that 3 minute session, now you're only charged for those active cycles. So if you're sitting at 10% utilization of CPU, you're only charged that 10%. And this is a massive change compared to a lot of, you know, existing container platforms. And something that's that's worth calling out where you're like, hey, I'm I'm on EC2 and I I get this instance type and it looks like it costs this much on cloud for containers, right? like there's a there's a big difference versus kind of traditional container solutions in terms of how that charging works specifically in regards to CPU. So, it's been a big change and and has made this uh a lot more affordable for for users which is great. We have to get into more hands. I I think that's it from our end. Like we're we're really excited about this product. There's been just a huge amount of interest here from all over in terms of how people are using this. The other thing I'll say, Kate, and and kudos to your team here, is like making it really easy to stick open code or cloud code in a box. We've got some really nice patterns there for like if that's how your end users are doing development, we've got some nice patterns for just like grabbing that and and running it as a background agent. And for anyone who is curious to have some, you know, out of the box examples, please do check out our GitHub. We have a bunch of ready to go um code pieces that you can just run on your local machine and see the product in in action immediately including the open code demo that Mike mentioned but there's a bunch of other things like collaborative terminal using our pdy features state machine using our R2 backups and a bunch of other things that directly demonstrate what you can do with sandboxes today. I think it's really cool that we are able just to give the agent this full development environment and get it as close as the human has it and kind of replicate the same loop with a bunch of tools um as like again terminal that you can connect a browser to. You can have a C interpreter with like persistent state. You can have background processes and you can have live preview URLs. You can have the file systems that emit the changes immediately and gives you feedback in real time. You can have egress proxy proxies for secure credential injections. And of course, you can have snapshot mechanism to make sure that the container wakes up instantly and you don't have to wait too long. This is really um really exciting project. Please do check up our npm and GitHub for more examples for code sources and and give it a try. Go and build cool things with it.