Stop Giving Your Agent Tools. Give It a Sandbox - Mikyo King, Arize AI

[music] >> Thanks for everyone for coming. So, my name is Mikail. I head up the open source group at Arise AI and we work on a fully open source observability and evaluation framework and we also work on our own agent. So, today we want I want to kind of walk you through what Code Mode is and our journey embracing Code Mode. So, if you look back in the last 30 years of software development, you could kind of segment software into two different categories. You could think of there's software built for software engineers and software for non-engineers. Take for example, you have you know, you're asked to get all the invoices out of your email inbox. If you're a programmer, you could write a script, [snorts] you can go in there, you can dynamically like see if there's an attachment, download it, put it into a folder. It might be a you know, 50 lines of code and you get this tool that you can use versus if you were a non-technical, you'd have to go into these inboxes, you have to manually download it. You're restricted by the interface of the software and you had to kind of buy that software. And the thing that's really interesting about things like Cloud Code is that that boundary is completely dissolving, right? Basically, anything you need, the coding agent can actually build that software for you

dynamically. And the labs and the foundation model providers know this, right? If you look at all the benchmarks like SWE-bench, that is what these models are being trained to do. And so, doesn't it make sense basically to align the agents you're building to be able to kind of build the software you need dynamically to dissolve that boundary. So, before I get to started in Code Mode, let's look at kind of what an agent looks like. So, hopefully you've built this uh uh kind of agent before. You give an agent kind of a huge catalog of tools, and then iteratively a user question comes in, it calls the tools that get pushed in the context window, it reasons about it, and then it calls another tool that get pushed in the context window. And hopefully most of you have built an agent like this. Uh certainly if you have a certain set of tools, this can work pretty well. But, we're now in that age where we want a lot more. We want much more emergent behavior. We want to put a lot more context into the agent.

We want to have longer horizons. And so, this is where you kind of get this bloat inside of the context. You'll see there's a lot of tools that get added into the context window that might not be relevant to the task. Um maybe the tool call is not responding with a lot of junk, or maybe needs to paginate through data. And so, you deal with this kind of context rot. You also deal with multi-turn, a lot of turns, as a result, and you're going to have to deal with compaction. So, luckily CodeMon solves a lot of these things. It's kind of an alternative architecture. Um instead of, let's say, just giving it a huge index or a menu of tools, you give it the ability to search for the specific tool, and then you also give it the ability to execute the tool. But, you don't give it just the ability to execute the tool, you actually give it a programming environment to execute those tools in different types of flows. So, they can do it sequentially, they can do them in parallel, um can put conditionals around it. Um and the nice thing about this is you can call multiple tools in one turn, and also return only what's necessary to push back into the context window of the model. And so, to kind of summarize it, put those these two things hypotheticals side by side, traditional tool calling, you know, you accumulate context window blow every time you call a tool. That's obviously going to, you know, add to latency and

cost. It's also going to increase increase turns. So, the time your user takes to get to value increases, right? So, has a lot of overhead. Um, on on the flip side, code mode potentially, when done well, can unlock a lot of abilities because it can call as many as it wants in a single turn and it only has to put in exactly what's necessary into the context window to solve its problem and you can also get a reduction in turn. So, that's sort of the the goal of of code mode. And so, let's just kind of go over some of the reasons why code mode works. Code mode works because one, you don't have to give it the giant catalog of tools up front. It can just pick and choose the tools that it needs and understand those tools and then execute just those. So, there's a progressive disclosure of tools. Uh, certainly there's things like tool search now also built into the foundation model providers. There's also the composability. There's a huge amount of benefit to not being able to being able to call tools in different sequences, either in sequence or parallel, fan out, chaining them together. You can create much more complicated and efficient things. If you've heard of obviously Claude's dynamic workflows is built on this composability but composability of sub agents.

I already mentioned sort of token efficiency. You don't have to spit out all the tools outputs and worry about the tools outputs being shoved into the context window. The model can dynamically kind of trim and search for certain areas and only put what it needs back into its context window. And the fourth one, which is the kind of the one of the major reasons we've embraced code mode, is this idea of emergent behavior, being able to build its own tools, right? So, for example, traditional with traditional tool calling, if I said for example like take all these records and remove personally identifiable information, it wouldn't be able to do that because it doesn't have a tool, or it might try to do it inside of its its context window, but then you're leaking credit card numbers in the context window, and that's not great. So, the emerging behavior of being able to create its own tools has a huge amount of benefit. So, here are some ways in which we've embraced Code mod. Uh so, we build our own agent uh and harness inside of Phoenix, our open source uh our observability platform, and we started with Bash. You know, Bash is a great tool of being able to compose together things like uh you know, said and grep to be able to uh compose together, you know, built-in Linux primitives with also our own CLI and be able to offload those into files for later use.

We also added um you know, a Python's sandbox, and we hoisted all our rest CRUD endpoints. Those are like perform getting traces or uh manipulating data sets, and we put that into this coding environment so that it can compose those create, read, update, delete operations, which unlocks like a whole category of things. And we also added one additional one, which is the ability to write SQL. And you know, text of SQL, you've obviously like heard like that probably many times before, but this also unlocks a lot of capabilities. The reason why is not everything fits into the tool category of just creating, updating, and deleting. Maybe you need to be able to do aggregate queries like tell me all the LM spans that have P95 latency of X, Y, and Z. You don't want to have to paginate through all your traces to do that. You want it the LM to be able to pinpoint find exactly what you're looking for. This is kind of a a neat and creative way we've also used code mode. So, we've actually also created a sort of a programming paradigm in our UI that's bound to our UI. So, we've been able to kind of be able to give it the ability to change the time range or be able to read data from the data store inside of the front end. And what's really cool about this is you kind of almost get this computer loose use like emergent behavior because we're able to put that programming paradigm in a web worker that has a two-way data binding directly into our browser and it can kind of do

complex UI interactions for you and it's really pretty fun to to watch it uh cook. So, code mode is not uh obviously for free. Um you're dealing with LLM generated code. If you've been at to the the Anthropic talks, you certainly probably seen some of these. You're dealing with resources. So, if you're working with hosted sandboxes, you have to worry about CPU, memory, how long those sandboxes stay around. Networking is a huge one. Maybe you want it to be able to go out to the web, but you never know what's out in the web. And so, you have to carefully think about network egress. Uh secrets is a hot topic. I think every sandbox provider is trying to come up with a different vaulting strategy. But, let's say you want to use Jev inside of your sandbox like Pi uh durable just shipped, you know, that's a crazy cool idea uh to be able to make fast decisions inside of a sandbox, but you have to make sure that those secrets don't come back into the context window.

And then the other one is permissions. Uh there's some very interesting research around this as well, but typically permissions are overbroad. And so, maybe in the first five sets the permissions are correct, but in the sixth operation it is not. Or maybe in composition uh the permissions are not. And so, you have to kind of think of guard railing some of these things if that is necessary. There's also operational complexity. Uh certainly sandboxes don't boot uh all the time perfectly. And so you have to you know be able to deal with failures. They're harder to debug because now you're debugging code and you're not debug debugging individual tools. And tool parameter accuracy becomes a much bigger issue because the tool parameter now is code. Uh so what if something fails in the middle? Uh tool discovery is also still an issue. Previously if you did have a tool, the LM would up front know it, but now it might have to go through a handshake to be able to understand which tools are in its catalog of things to be able to do.

And then this is sort of a big one, you know, you see code mode being injected into MCP, but then you see a lot more code mode being pushed also into harnesses. And now you have a double sandboxing uh problem. So sandboxing, you know, is going to return stuff out as serialized JSON and then it needs to be parsed and then put into another code mode. And so there's this like double duty that ends up happening that also can be on operational complexity. These are sort of open problems. Uh they're not like blockers, but durability is a problem. So let's say you build a giant piece of code that needs to execute and it breaks in the middle. You can't resume from that step. Um if determinism and mathematics are important to you, for example, it might actually force you to think about which programming language you're using. So JavaScript is something LLMs are really good at doing, but maybe not great for math. Um For example, uh code mode can be great way to do multimodal and manipulating images or creating bytes, but getting that out of the sandbox is quite tricky, right?

Because you don't want to serialize the whole binary, the whole image. That's going to end up in the the context window. So you might have to do some sort of signed image handoff. And then the other one which is like the Frontier models are amazing at code mode, but if you're trying to do cost saving and you want to swap out a smaller model, its capabilities might not just quite um get there. So, how do we at Arize evaluate code mode? Um we use our own We dogfood our own platform, which is called Arize Phoenix, um and we run our agents through various permutations through something called the Harbor framework, which basically puts the agents into uh sandbox environments and runs them on clear instructions with network access turned off, and we run this multiple times in trials, so we eliminate a lot of the flake. And we build evals in the form of Harbor verifiers to make sure that the agent's doing the correct thing. And then, Harbor's sort of rooted in this RL framework. We give it a a one or a zero to indicate whether or not performs the task, and we test it across many different permutations.

So, this is kind of why you want to test like this, because as I mentioned, you can get a lot of flake. Um sometimes it might work where you're dealing with an indeterminate system. It might take different trajectories uh depending how you do it, and we uh track pass at K of each individual task that we think it should pass. So, you know, how does this work? Uh this is kind of our results from uh running this Harbor run. You can see pre uh code mode and post code mode, we, you know, decreased the amount of tokens by 43%. The cost went down by 31%, and you know, there was some overhead with the sandbox, but the latency stayed pretty constant. So, we knew that across these multiple surfaces, whether or not it was cloud code or MCP or the CLI, we had accomplished something that uh was definitely better than it was before. How do we use this data? It's really uh a huge unlock if you have you both capture the trajectory and the evals. As you've seen with a lot of the other talks, this gives you a very great environment to reinforce and look at the behavior of these agents and be able to tell it how to hill climb. So, maybe, you know, first time we introduce a new tool, sometimes actually the efficiency goes down. And so, we can tell a coding agent why in these specific scenarios, what parts of the instruction are are guiding it in the wrong direction and we're able to hill climb these evals to be able to make our agents better.

So, if you don't take anything other than this away, hopefully this will sum it up. We really like code mode at RI because of the fewer turns you get out of code mode. We want you to be able to bring any model for us. So, we want it to be cheaper and we're really excited about the emergent behavior and the long horizon tasks that our agents can do. And so, I hope this talk convinces you to give your agent a sandbox and then use one of our tools to evaluate it. And that's it. Thank you.