Give Your LLM a Sandbox: Introducing Dynamic Isolated Workers

During developer week, we introduced this concept of MCP code mode. In traditional MCP, you provide a set of tools or functions to the context which an LLM can call. Um, it's a great implementation, but it doesn't always work for a couple of reasons. One, it's quite unreliable. That's because the LLMs don't have enough training data on making two calls. Two, it bloats the context of the LLMs. And three, when you need to do multi-step tool calls such that the response from one to two call is used in the second to call and then in the next, it doesn't work. It completely breaks. And that's because there's a lot of context switching and LLMs just can't handle it. So we introduced this new concept of code mode. Code mode essentially converts MCP tools into a simple reliable TypeScript API that the LLM can write code against and all of that is securely executed in a sandbox. This works really well because LLMs are great at writing code. They've seen billions of billions of lines of code. So when it comes to writing code to achieve the objective, they are really good at doing this. And for complex two calls, this can be simply expressed as code and the final result is uh returned to the LLM after it's all been executed. And of course, this in no way bloats the context. So it's a great implementation for complex tool use. Code mode is available today and you can go give it a try out. I'm going to have it linked in the description of this video so you can go check it out. But more importantly, code mode is powered by a dynamic sandboxing technology called the dynamic worker loader API. So now I'm sure you're wondering what is the dynamic worker loader API. To put it simply, it's a binding that lets you create a new worker context on the fly to run any arbitrary code. And the reason why it's powerful is because your worker can spin up several child workers in their own isolated context with their own environments and bindings or capability to run code securely. So instead of talking more about what it is, I'm actually just going to show you what it is. So let's uh switch to my computer and I'll show you what this look like. I'll show you what a dynamic worker is and then we'll look at a few features that this new API has. So, switching on to my screen. Um, here is an example of what a dynamic worker looks like. Um, I have it in the binding env.loader. And of course, to get this up and running, you have to set up the binding. So, if I go to my Wrangler.json file, um, I have the worker loaders binding and I have a binding called loader. You can call this whatever you want that um automatically creates that context for the worker to run in. And um that's what you have here. Then [snorts] the next thing you do is you pass it an ID. This is a unique ID that creates a new isolate context for the worker if one does not already exist. Which means if you already have a child walker with the same ID, uh the request is just going to be sent to that worker and you don't have to create a new context each time. So it's reusable in that sense. And of course uh this is the callback that describes the code that is executed inside of the new worker context. So you can think of this return as your wrangler.json because this is how you configure the child walker. So you have a compatibility date. You tell it what the main model is and uh this is the content of the main model which is just a JavaScript file. Um this is the code in the context and that's it. But the cool thing is that you can do things like set up the environment. For example, I have a my var variable which in turn is then used inside of the worker. So I'm returning hello plus the content of that variable. And uh because this is was actually designed to be a sandbox environment, especially when it comes to running code securely generated by LLMs, you can choose to turn off internet access. And then uh the code being executed in this sandbox has no access to the outside internet. So let's uh give this a spin and I'll just start up this worker by running the dev script. And uh I'm just going to make a coral coral request to my worker. And you see that this returns hello world. That's because um the script here is configured to return hello plus whatever it is the content of that variable is which is super cool. So let's go on and let me show you a few other things you can do here with this dynamic worker loader API. I I have an example that um shows you how you can customize the internet access when it comes to what this sandbox has access to. So, let's go look for outbound.ts. I'm just going to copy this and let's put it in the index file. Okay, that looks good. Um so, let me show you what you can do here. Um, in addition to just completely blocking off the um, internet from your sandbox, you can actually write a proxy function to customize what happens when this sandbox tries to access the internet, which if you ask me is really powerful. So, let me show you what uh, this is able to do. First, I'm using the context.export feature. Again, you have to get this enabled by setting the setting it in your parents workers compatibility flag. Enable context to export. This enables you create uh custom functions that the worker itself can use. So in this case, we're creating a proxy function uh which has a method called fetch. And whenever a fetch request is made inside of my sandbox, um I just want to return the content of what's inle.com. So you could do whatever it is you want here. You can redirect to a different origin. You can block certain origins. Um that's if you want your the code inside of your sandbox to have access to the internet, but you only want it to be able to talk to predefined uh origins. You can like rewrite headers, do whatever you want to do, whatever it is a worker can actually do. So, take a look at this. The content of my uh sandbox sandbox script is trying to make a request to an insecure site. And what I'm doing here is just blocking it and returning the uh content of example.com. So, I'll give this a save. And uh let's make a call request to that endpoint. And of course, as you'd expect, this returns the response from the example.com website, which gives you lots of flexibility. You could allow the internet, you could block it, you could customize the behavior of the internet behavior in a sandbox. It's all really good and fun. The last thing I'm going to show you here is um the last thing I'm going to show you here is setting up custom bindings uh for your sandbox which in a way is how you securely give your sandbox capabilities to do things. So instead of providing an API token or an O token to your sandbox for it to make its own request, you could write your own cleanly implemented uh uh custom uh worker entry point class that has that ability to carry out that um action and then provides that as a capability to your sandbox. So here's an example of it. I have that in um bindings.ts. TS. Okay, we'll just copy this and let's take a look at what this looks like. So, um I have it's the same worker you saw earlier. Very very similar. In the environment, I have this new capability called greeter. And to set this up, I'm passing passing along some props, which is just a text called Alice. Uh you can think of this props as a secure way to pass things like API secrets or oat tokens to the custom uh binding. And here's the implementation of the custom binding. In this case, it's just returning hello and the name we have passed down to the props. It has no idea what the actual value is, which is cool. But even more importantly, the worker itself which is running in the sandbox has no access to the context of the greater function and also the secrets it has um that it has received through props. It only knows how to call it and perhaps pass parameters to it but has no idea what its implementation is and other secrets it may contain which is a really cool way to give permissions and capabilities to your worker running inside of a sandbox. So, I'll give this a save and let's uh let's go run this. Okay, I don't think I copied that to the right file. So, let's go back up here. So, I'll give this a save. Let's head back here and I'll make a call request to that worker. And you can see that this returns hello Alice and Alice is coming from the props that has been given to this custom Britta function. And uh I think it's really cool. I think the dynamic worker loader API is really powerful especially it's also useful when you need to securely execute codes generated by LLMs and then present it with permissions through bindings. Kenton did a 40inut talk on this. So, I'll encourage you go check it out. I'll have it linked in the top corner of this video so you can go watch it and see for yourself. The dynamic worker loader API is in private beta right now. I'm going to have a link below if you like to sign up to test this in workers right now. Uh, but you can also try this locally in Wrangler today just as you saw in this tutorial. We love to get your feedback on this new enhancement to the workers platform. So, let me know your thoughts on this in the comments and I will see you in the next one. Take care. Bye. [music]