Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4) — Anthropic

Okay, we have a Okay, we have a question here. question here. Greetings. How do you see Greetings. How do you see CICD for Convex agents? Hmm CICD for Convex agents? Hmm . . . Ah, that's a great Ah, that's a great Ah, that's a great question. We have question. We have question. We have something called NCLI, which is a something called NCLI, which is a something called NCLI, which is a CLI that essentially CLI that essentially CLI that essentially wraps our API for wraps our API for wraps our API for calls and allows you to calls and allows you to calls and allows you to do things similar to do things similar to do things similar to Terraform where you pass in Terraform where you pass in Terraform where you pass in YAML. So you can YAML. So you can YAML. So you can put your put your put your long-lived resources long-lived resources long-lived resources in YAML format in your in YAML format in your in YAML format in your codebase and then codebase and then codebase and then build CICD to build CICD to build CICD to update NCLI update NCLI update NCLI with that YAML with that YAML with that YAML whenever you whenever you whenever you make changes to make changes to make changes to your resource. Is that your resource. Is that your resource. Is that clear? clear? clear? Got it. So I Got it. So I Got it. So I assume you assume you assume you guys guys guys use the same thing use the same thing use the same thing internally internally internally . . . Yes, that's right, if Yes, that's right, if Yes, that's right, if we need to we need to we need to create reliable create reliable create reliable long-lasting resources long-lasting resources , we can simply , we can simply , we can simply lock them in. lock them in. lock them in. Okay, thank you. Okay, thank you. Okay, thank you. Thank you. Thank you. Thank you. I think you mentioned I think you mentioned I think you mentioned that you can send that you can send that you can send messages at messages at any time. Can I any time. Can I any time. Can I send send send a message if a message if a message if the model is waiting for the the model is waiting for the the model is waiting for the result of a result of a result of a special special special tool?

tool? tool? Not when she is waiting Not when she is waiting Not when she is waiting for the result of a for the result of a for the result of a special special special tool. We are tool. We are tool. We are actually actually actually pausing pausing pausing execution at this time. execution at this time. execution at this time. But if the model But if the model But if the model is doing other work, is doing other work, is doing other work, then we allow then we allow then we allow you to you to you to queue messages. queue messages. queue messages. However, when it However, when it However, when it processes the result of a processes the result of a processes the result of a special special special tool, we tool, we tool, we expect you to expect you to expect you to send it; therefore, send it; therefore, send it; therefore, for for for consistency reasons, we consistency reasons, we consistency reasons, we block calls. block calls. block calls. Well, what happens Well, what happens Well, what happens if I send if I send if I send the message anyway? the message anyway? the message anyway? You will get a 400 error. You will get a 400 error. Okay, that makes sense. Okay, that makes sense. There was a request There was a request There was a request to show the slide with the QR to show the slide with the QR code again. code again. code again. The one that... The one that... The one that... The one with the credits, The one with the credits, The one with the credits, yes. yes. Are there any more questions? Are there any more questions? If you're shy, If you're shy, If you're shy, we... we... we... Yes, we'll stay Yes, we'll stay Yes, we'll stay after we're done after we're done after we're done if you have any if you have any if you have any questions that you don't questions that you don't questions that you don't want... want... I'm browsing the I'm browsing the repository. There are repository. There are repository. There are many interesting many interesting many interesting reports from reports from reports from workshops. Is workshops. Is workshops. Is there anything missing, like there anything missing, like there anything missing, like slides that slides that slides that would help us would help us would help us get through the other... get through the other... get through the other... Yes, yes. ... Yes, yes. ... sessions like sessions like sessions like this, are they completely self- this, are they completely self- this, are they completely self- explanatory thanks to the explanatory thanks to the explanatory thanks to the readme files, because you were readme files, because you were readme files, because you were running a running a running a skills-building workshop.

skills-building workshop. skills-building workshop. Yes, yes. Other Yes, yes. Other Yes, yes. Other workshops were from our workshops were from our workshops were from our Code with Cloud conference. Code with Cloud conference. Code with Cloud conference. Most of them Most of them Most of them can be found online can be found online can be found online in the version we in the version we in the version we ran in London ran in London ran in London about a month ago. about a month ago. about a month ago. Yes, all those talks are Yes, all those talks are Yes, all those talks are on YouTube if you're on YouTube if you're on YouTube if you're interested in other interested in other interested in other repositories, but this one repositories, but this one repositories, but this one should give you a pretty should give you a pretty should give you a pretty good overview, and it's good overview, and it's good overview, and it's very similar to what very similar to what very similar to what we've done at other we've done at other we've done at other conferences. conferences. Yes. Yes. So, going back to So, going back to So, going back to assessment, can assessment, can assessment, can you talk about how you you talk about how you you talk about how you maintain quality maintain quality maintain quality and best practices and best practices and best practices around around around assessment tools? assessment tools? assessment tools? Yes, we do Yes, we do Yes, we do a lot of evaluations, a lot of evaluations, a lot of evaluations, especially as especially as especially as new models and new models and new models and new capabilities emerge. new capabilities emerge. new capabilities emerge. Of course, we have Of course, we have Of course, we have access to the models access to the models access to the models before you do. So, before you do. So, before you do. So, one of the things we one of the things we want to ensure as a managed service want to ensure as a managed service is to is to is to update our update our update our assumptions about assumptions about assumptions about tools as early as possible to tools as early as possible to tools as early as possible to remove what is no longer remove what is no longer remove what is no longer needed. And we needed. And we needed. And we do all this based on do all this based on do all this based on evaluation. We have evaluation. We have evaluation. We have internal internal internal tools that tools that tools that work with different work with different work with different datasets to datasets to datasets to run our sessions. run our sessions.

Since everything is in the cloud, Since everything is in the cloud, it's easy to run, it's easy to run, it's easy to run, say, 10,000 sessions say, 10,000 sessions say, 10,000 sessions to do all of to do all of to do all of this in parallel. this in parallel. this in parallel. We then analyze We then analyze We then analyze the results, the results, the results, adjust adjust adjust the prompts and the prompts and the prompts and assumptions, and assumptions, and assumptions, and remove or remove or remove or add elements to add elements to add elements to ensure the best ensure the best ensure the best performance with each performance with each performance with each model release. We model release. We model release. We evaluate evaluate evaluate a wide variety of a wide variety of a wide variety of things for general things for general things for general purposes, not just purposes, not just purposes, not just programming. programming. programming. So how does this apply So how does this apply So how does this apply to the agents we to the agents we to the agents we create? Is there a create? Is there a create? Is there a recommended way recommended way recommended way to evaluate, or to evaluate, or to evaluate, or should I should I should I use my use my use my own solutions? own solutions? own solutions? I think the best I think the best I think the best advice we can advice we can advice we can give is to give is to give is to use use high-quality data for evaluation. high-quality data for evaluation. I think Claude Code does I think Claude Code does I think Claude Code does a good job of generating a good job of generating a good job of generating estimates for you, but estimates for you, but estimates for you, but garbage in is garbage in is garbage out. We garbage out. We garbage out. We noticed that some of noticed that some of noticed that some of our initial ideas our initial ideas our initial ideas about grading were about grading were about grading were a bit vague: you a bit vague: you a bit vague: you get 100%, but what does get 100%, but what does get 100%, but what does that mean for that mean for that mean for performance? Therefore, it is performance? Therefore, it is performance? Therefore, it is worth working on worth working on worth working on the data and the data and the data and making sure that you are making sure that you are making sure that you are setting setting setting the task correctly. I know this isn't an out- the task correctly. I know this isn't an out- the task correctly. I know this isn't an out- of- of- of- the-box solution, but the-box solution, but the-box solution, but it's usually the best it's usually the best it's usually the best advice that works.

advice that works. advice that works. You can You can You can review your review your review your past sessions, past sessions, past sessions, the results you got, the results you got, the results you got, try try try to evaluate them, and see to evaluate them, and see to evaluate them, and see how the model works. This would be a how the model works. This would be a how the model works. This would be a good good good way to create a way to create a way to create a dataset dataset dataset based on the analysis of your based on the analysis of your based on the analysis of your historical sessions. historical sessions. I have a question I have a question here. here. here. Hello, thank you for the Hello, thank you for the Hello, thank you for the interesting information. interesting information. interesting information. I have a question about how to I have a question about how to I have a question about how to share such share such share such agents and all agents and all agents and all resources within an resources within an resources within an entire organization. I entire organization. I entire organization. I ask because we ask because we ask because we often use often use often use Claude as a code. We are Claude as a code. We are Claude as a code. We are creating shared creating shared creating shared skills. We build skills. We build skills. We build common connectors. And common connectors. And common connectors. And with everything you with everything you with everything you shared today, shared today, shared today, like the agent, the like the agent, the like the agent, the memory store, or the memory store, or the memory store, or the MCP channel, can MCP channel, can MCP channel, can we actually we actually we actually share them or share them or share them or partially distribute partially distribute partially distribute them across the organization between them across the organization between them across the organization between different departments? different departments? different departments? So, what are the So, what are the So, what are the limits? What privileges limits? What privileges limits? What privileges are granted for these are granted for these are granted for these resources? Are resources? Are resources? Are these these these resources tied to my resources tied to my resources tied to my account, or account, or account, or how can I how can I how can I control this? So, control this? So, control this? So, what, what's the way to what, what's the way to what, what's the way to do it? do it? do it? They are tied to They are tied to They are tied to your your your workspace. So workspace. So any API key within any API key within any API key within your your your workspace can workspace can workspace can access the resources access the resources access the resources that are that are that are mine, mine, mine, can my teammates can my teammates can my teammates edit edit edit or change what I or change what I or change what I create, is that, create, is that, create, is that, fully fully fully controlled, right?

controlled, right? controlled, right? I think you have a I think you have a Claude platform administrator, so Claude platform administrator, so you will have an organization you will have an organization you will have an organization for your company. And for your company. And for your company. And then your then your then your administrator will administrator will administrator will configure the configure the configure the workspaces and workspaces and workspaces and decide who will decide who will decide who will have access to have access to have access to which space. which space. which space. Yes. I think you Yes. I think you would basically design would basically design workspaces based on workspaces based on workspaces based on your your your access control. access control. access control. Right. Right. Right. You'll probably want to You'll probably want to You'll probably want to do this: whatever do this: whatever do this: whatever group of people you group of people you group of people you want to allow access to want to allow access to want to allow access to X resources, put X resources, put X resources, put them in one space. them in one space. them in one space. So you can So you can So you can share share share anything, even anything, even anything, even partially, you know, you partially, you know, you partially, you know, you create an agent create an agent create an agent that can do something that can do something , but I , but I , but I only want to share only want to share only want to share part of it. part of it. part of it. For example, I share For example, I share For example, I share a memory with someone, and the a memory with someone, and the a memory with someone, and the MCP channel with others. MCP channel with others. MCP channel with others. Is this possible? Do I Is this possible? Do I Is this possible? Do I need to share the need to share the need to share the entire technical entire technical entire technical stack of what I'm stack of what I'm stack of what I'm building? building? building? We don't have the We don't have the We don't have the fine-tuning fine-tuning to share this or to share this or to share this or not share this, but not share this, but not share this, but you as a developer and you as a developer and you as a developer and integrator will be able to integrator will be able to integrator will be able to control this in control this in control this in enough detail that enough detail that enough detail that your end your end your end users have users have users have access to partial access to partial sharing. sharing. Steeply. Perfectly. Steeply. Perfectly. Steeply. Perfectly. Thank you very much. Thank you very much. Thank you very much. Thank you. Thank you. Thank you. Yes. Yes. Yes. So, essentially, we So, essentially, we So, essentially, we can generate can generate can generate a lot of code a lot of code a lot of code using a lot of using a lot of using a lot of agents, and then we agents, and then we agents, and then we hit a bottleneck hit a bottleneck hit a bottleneck : uh, looking at : uh, looking at : uh, looking at that code, that code, that code, making sure you making sure you making sure you can actually can actually can actually merge it. Um, what do you merge it. Um, what do you merge it. Um, what do you guys think about guys think about guys think about this? Are you deploying this? Are you deploying this? Are you deploying something, because I see you something, because I see you something, because I see you do it every 30 do it every 30 do it every 30 days, which might be a days, which might be a days, which might be a different definition of different definition of different definition of deployment, but is deployment, but is deployment, but is it deploying it deploying it deploying something that you can something that you can something that you can actually test actually test to make sure to make sure to make sure it works well, or it works well, or it works well, or how do you review, uh, how do you review, uh, so many so many so many lines of code from all the lines of code from all the lines of code from all the agents?

agents? agents? So the original So the original So the original seminar that we seminar that we seminar that we wanted to do wanted to do wanted to do was actually aimed at was actually aimed at was actually aimed at answering this answering this answering this question. How do you, how do you question. How do you, how do you question. How do you, how do you create agents create agents create agents to review code? to review code? to review code? Um, we found that you Um, we found that you Um, we found that you can create can create can create an integration with CMA so that an integration with CMA so that an integration with CMA so that during PR it during PR it during PR it runs one, two, runs one, two, runs one, two, three, many sessions, three, many sessions, three, many sessions, many subagents many subagents many subagents to assess what to assess what to assess what has changed and find has changed and find has changed and find errors. Internally, errors. Internally, errors. Internally, we use we use we use these types of agents to these types of agents to these types of agents to do a significant do a significant do a significant amount of work, and we've amount of work, and we've amount of work, and we've seen that it seen that it seen that it allows us to allows us to allows us to reduce the time per reduce the time per reduce the time per PR, reduce PR, reduce PR, reduce errors, and errors, and errors, and maintain high maintain high maintain high quality. But this is quality. But this is quality. But this is partly an open partly an open partly an open question, since question, since question, since we have now removed we have now removed we have now removed the restrictions on the restrictions on the restrictions on printing speed and printing speed and printing speed and significantly increased significantly increased significantly increased the number of PRs we the number of PRs we the number of PRs we can do per day can do per day , hour, or second. , hour, or second. How can we keep up? We're How can we keep up? We're still researching this still researching this , but , but , but yeah. yeah. yeah. I think the CMA is I think the CMA is I think the CMA is part of the answer. part of the answer. part of the answer. Yeah, and you can Yeah, and you can Yeah, and you can iterate in a loop, you iterate in a loop, you iterate in a loop, you know, like, know, like, know, like, test, add test, add test, add comments, comments, comments, review until it's review until it's review until it's ready, but at ready, but at ready, but at some point you some point you some point you still need to still need to still need to run it to run it to run it to make sure it's make sure it's make sure it's working well working well working well locally. So, I think locally. So, I think this is where deployments can this is where deployments can this is where deployments can help help help , where you , where you , where you can, you know, can, you know, can, you know, automatically automatically automatically deploy something that deploy something that deploy something that can be tested can be tested can be tested using using using your own your own your own deployment agent, like you deployment agent, like you deployment agent, like you said. It just said. It just said. It just depends on whether depends on whether depends on whether you have different you have different you have different environments for environments for environments for each branch, but each branch, but each branch, but you can you can you can also run your app in a also run your app in a also run your app in a sandbox and sandbox and sandbox and let Claude let Claude let Claude independently independently independently check out the check out the check out the changes he makes. And changes he makes. And changes he makes. And make sure it make sure it make sure it also acts as a also acts as a also acts as a quality control agent. Are quality control agent. Are

quality control agent. Are these long-term these long-term these long-term sessions that you can sessions that you can sessions that you can go into and actually go into and actually go into and actually work with the work with the user interface? user interface? We don't have We don't have We don't have that that that tunneling capability yet, tunneling capability yet, tunneling capability yet, so you could so you could so you could let Claude go in let Claude go in let Claude go in there and work with the there and work with the there and work with the browser. browser. browser. That's right. And then That's right. And then That's right. And then check it out. And check it out. And check it out. And then maybe then maybe then maybe even provide you with a even provide you with a even provide you with a video, as an option. video, as an option. video, as an option. Yes, exactly. Yes, exactly. Yes, exactly. Steeply. Thank you. Steeply. Thank you. Are there any more questions? Are there any more questions? Greetings. Greetings. Greetings. Congratulations. Congratulations. Congratulations. For an enterprise For an enterprise For an enterprise company, if we company, if we company, if we want to support want to support want to support true true true tenant isolation, I heard that tenant isolation, I heard that tenant isolation, I heard that you probably don't you probably don't you probably don't support support support use cases for use cases for use cases for enterprise enterprise enterprise companies yet. companies yet. companies yet. We do not support We do not support on-prem hosting. on-prem hosting. Local? Local? Local? Yes. Yes. Yes. But what if I want to But what if I want to But what if I want to have, say, 100 have, say, 100 have, say, 100 clients as a clients as a clients as a corporate corporate corporate company? Can I company? Can I company? Can I have separate managed have separate managed have separate managed instances of Claude agents? instances of Claude agents? Yes. Yes. for each of our for each of our for each of our companies that we companies that we companies that we want to support? want to support? want to support? Is this a viable Is this a viable Is this a viable option that you option that you option that you support? Do we support? Do we support? Do we only get one only get one only get one copy, or can we copy, or can we copy, or can we get one get one get one copy for each copy for each copy for each company we company we company we support as a support as a support as a corporation?

corporation? corporation? It depends on It depends on It depends on how you choose to how you choose to how you choose to define it. You can define it. You can define it. You can define an agent for define an agent for define an agent for each client and each client and each client and then the data will be then the data will be then the data will be isolated, or you isolated, or you isolated, or you can can can use a use a use a workspace for workspace for workspace for each client, if each client, if each client, if you want true you want true you want true isolation you can isolation you can isolation you can create an organization create an organization create an organization for each client. for each client. for each client. So it depends on So it depends on So it depends on how you want how you want how you want to configure the system. to configure the system. to configure the system. Your agents, and you will Your agents, and you will Your agents, and you will probably want probably want probably want to maintain a to maintain a to maintain a representation of which representation of which representation of which client is associated with which client is associated with which client is associated with which agent. agent. agent. Good. I think I have an Good. I think I have an Good. I think I have an additional question. additional question. So, given So, given the scenario where we have the scenario where we have the scenario where we have two clients and we two clients and we two clients and we want to create an want to create an want to create an agent once, but have it agent once, but have it used by used by both clients. How both clients. How both clients. How to achieve this? to achieve this? to achieve this? There are many ways to There are many ways to There are many ways to do this. You can do this. You can do this. You can do this through do this through automation. You automation. You can store can store can store your your your agent definition in YAML format in agent definition in YAML format in agent definition in YAML format in code and then code and then code and then create tools create tools create tools to update it in to update it in to update it in both both both workspaces and essentially workspaces and essentially workspaces and essentially synchronize them, synchronize them, synchronize them, or you can or you can or you can model everything model everything model everything within a single within a single within a single organization. Uh-huh.

organization. Uh-huh. organization. Uh-huh. You could create a You could create a You could create a single agent for single agent for single agent for all your clients all your clients all your clients and then manage. and then manage. and then manage. By showing By showing By showing which client which client which client is using which is using which is using which session. session. session. So, then in which So, then in which So, then in which workspace workspace workspace can I create this can I create this can I create this agent? agent? Yes, if you do this Yes, if you do this for each for each for each workspace, then we workspace, then we workspace, then we provide tools provide tools provide tools to synchronize a to synchronize a to synchronize a single definition in single definition in single definition in your code with your code with your code with multiple places so that multiple places so that multiple places so that every time you every time you every time you update it via update it via update it via CI, you can update CI, you can update CI, you can update all the definitions all the definitions all the definitions at once. at once. at once. So you can either So you can either So you can either make a copy of your make a copy of your make a copy of your agent, which is essentially a agent, which is essentially a agent, which is essentially a template of the system template of the system template of the system query, model name, query, model name, query, model name, and tools, and and tools, and and tools, and make a copy for make a copy for make a copy for each each each workspace. Let's say workspace. Let's say workspace. Let's say you choose the " you choose the " one workspace one workspace one workspace per client" option, or you per client" option, or you per client" option, or you can simply can simply can simply put all your put all your put all your clients in one clients in one clients in one workspace and workspace and workspace and then maintain a then maintain a then maintain a reflection of whether reflection of whether reflection of whether you create copies of you create copies of you create copies of your agent or your agent or your agent or use a use a use a single agent. single agent. single agent. Showing which Showing which Showing which session belongs to which session belongs to which session belongs to which client. client. client. We have a metadata field We have a metadata field We have a metadata field in the session where you can in the session where you can in the session where you can insert, say, a insert, say, a client ID, so you know client ID, so you know which session belongs to whom which session belongs to whom which session belongs to whom . Client.

. Client. Okay, if we Okay, if we put all put all put all customer data in one customer data in one customer data in one workspace, how do workspace, how do workspace, how do we satisfy we satisfy we satisfy the requirement for the requirement for the requirement for data separation? data separation? Yes. Yes. A requirement, you know. A requirement, you know. A requirement, you know. Yes, if you are concerned about Yes, if you are concerned about Yes, if you are concerned about data separation, then data separation, then data separation, then you need to you need to you need to place it in place it in place it in separate separate separate workspaces, and resources workspaces, and resources workspaces, and resources are allocated for are allocated for are allocated for each each each workspace. But, in essence workspace. But, in essence , there are tools that , there are tools that , there are tools that allow you to allow you to allow you to do each of these do each of these do each of these things. things. things. And when you talk And when you talk And when you talk about data separation, is about data separation, is about data separation, is that a legal requirement or is that a legal requirement or is that a legal requirement or is it just something you it just something you it just something you want? want? want? Yes, therefore. Yes, therefore. Yes, therefore. Something legal, Something legal, Something legal, like a client doesn't like a client doesn't like a client doesn't want to share data want to share data want to share data with other clients, with other clients, with other clients, you know, they want you know, they want you know, they want dedicated space. dedicated space. dedicated space. Yes, you would do this Yes, you would do this Yes, you would do this for each for each for each workspace. workspace. workspace. Good. OK. Thank you. Good. OK. Thank you. Good. OK. Thank you. I saw the question here I saw the question here . You need to . You need to . You need to speak louder, speak louder, speak louder, or or Um, is there a way Um, is there a way to change the to change the to change the computing computing computing environment, i.e. in environment, i.e. in environment, i.e. in some cases I some cases I some cases I might need might need might need more memory for more memory for more memory for my agent to run, my agent to run, my agent to run, or I need a GPU or I need a GPU or I need a GPU for it to be able to for it to be able to for it to be able to perform tasks. perform tasks.

Is this supported Is this supported here? here? here? Currently, if you Currently, if you Currently, if you want to configure want to configure want to configure resource allocation resource allocation resource allocation for your sandbox for your sandbox , you should , you should , you should use your use your use your own hosts. You own hosts. You own hosts. You can can can use something use something use something like Modal, which has like Modal, which has like Modal, which has GPU support. We have GPU support. We have GPU support. We have guides and a guides and a guides and a kind of native kind of native kind of native integration so you integration so you integration so you can hook up can hook up can hook up your Modal instances your Modal instances your Modal instances to run the agent workflow to run the agent workflow inside Modal instead of inside Modal instead of our "sandboxes" our "sandboxes" our "sandboxes" which are all the which are all the which are all the same these days. So if same these days. So if same these days. So if you are interested in a different you are interested in a different you are interested in a different scale, you will scale, you will scale, you will use a use a use a different different different provider for that. provider for that. provider for that. And do you have a list of And do you have a list of And do you have a list of specifications of what specifications of what specifications of what these agents have access to these agents have access to these agents have access to in the environment? in the environment? in the environment? Yes, you can find Yes, you can find Yes, you can find this in our this in our this in our documentation, as well as documentation, as well as documentation, as well as guides for guides for guides for integration with integration with integration with third-party third-party third-party providers. providers. providers. Yes, we have a whole Yes, we have a whole Yes, we have a whole bunch of guides on bunch of guides on bunch of guides on how to work with Modal, how to work with Modal, how to work with Modal, EB2, Daytona, or any EB2, Daytona, or any EB2, Daytona, or any other other other sandbox provider. Yes. sandbox provider. Yes. sandbox provider. Yes. Greetings. Greetings. Greetings. Greetings. Greetings. Greetings. Thank you again. So, out of Thank you again. So, out of Thank you again. So, out of curiosity, how do curiosity, how do curiosity, how do you create a similar you create a similar you create a similar version of Cloud Tag version of Cloud Tag version of Cloud Tag using CMA in another using CMA in another using CMA in another chat channel, especially chat channel, especially chat channel, especially for for multiplayer?

This can certainly be This can certainly be done today: done today: done today: you can map you can map you can map each thread to a session each thread to a session , and every time a , and every time a , and every time a user writes a user writes a user writes a message, you message, you message, you add it as add it as add it as user.message so the agent user.message so the agent user.message so the agent knows, "Hey, this is from knows, "Hey, this is from knows, "Hey, this is from Gabe." Or if it's Gabe." Or if it's Gabe." Or if it's from Priyanka. So from Priyanka. So from Priyanka. So anyone can write to anyone can write to anyone can write to this agent this agent this agent independently. You can independently. You can independently. You can set up a shared set up a shared set up a shared memory for sessions so that memory for sessions so that memory for sessions so that it learns from it learns from it learns from the experiences of others. You the experiences of others. You the experiences of others. You can give him can give him can give him the tools to the tools to the tools to respond, for example respond, for example respond, for example via a Slack post or something via a Slack post or something via a Slack post or something similar, so that the agent similar, so that the agent similar, so that the agent decides when decides when decides when to post to post to post the message. You don't the message. You don't the message. You don't have to just have to just have to just duplicate everything the duplicate everything the duplicate everything the agent responds to. You agent responds to. You agent responds to. You want it to be want it to be want it to be obvious. But you obvious. But you obvious. But you can certainly can certainly can certainly model this model this model this with CMA and with CMA and with CMA and And if you want to And if you want to And if you want to do it as a Cloud Tag, do it as a Cloud Tag, do it as a Cloud Tag, you would most likely you would most likely you would most likely do one do one do one agent per channel in Slack, agent per channel in Slack, agent per channel in Slack, and then each agent would and then each agent would and then each agent would have separate memory have separate memory have separate memory and access only to and access only to and access only to messages in that messages in that messages in that channel. channel. Yes. Yes. Hello, thank you for Hello, thank you for Hello, thank you for the presentation. Very the presentation. Very the presentation. Very useful. Quick useful. Quick useful. Quick question regarding question regarding question regarding loops and long loops and long loops and long runs in this Cloud runs in this Cloud runs in this Cloud Console, how do you see it? Is Console, how do you see it? Is there some special module for this that I can't find, that I can't find, or do you plan or do you plan or do you plan to develop this in the to develop this in the to develop this in the future? Thank you. Do future? Thank you. Do you mean how do you mean how do we provide we provide we provide observability observability observability for very long sessions for very long sessions for very long sessions or or or No, no, no. Um, more of a No, no, no. Um, more of a No, no, no. Um, more of a focus on focus on focus on using this

using this using this platform to platform to platform to create these create these create these long-acting agents long-acting agents long-acting agents that work for hours, that work for hours, that work for hours, like 6 hours. Is like 6 hours. Is like 6 hours. Is this already built into what this already built into what this already built into what you've built into you've built into you've built into agents, or am I agents, or am I agents, or am I just curious just curious just curious to know if to know if to know if you're developing something you're developing something you're developing something specifically for this, specifically for this, specifically for this, or how you see this or how you see this or how you see this trend. The trend. The results system results system I demonstrated is the I demonstrated is the I demonstrated is the best way best way best way for very long sessions for very long sessions , because you , because you , because you define the define the define the exit criteria and Claude will exit criteria and Claude will exit criteria and Claude will continue working continue working continue working until it meets those until it meets those until it meets those criteria. And it could criteria. And it could criteria. And it could be a very long be a very long be a very long task, like task, like task, like rebuilding something, and rebuilding something, and rebuilding something, and he would he would he would repeat it tirelessly until he repeat it tirelessly until he repeat it tirelessly until he achieved achieved achieved compliance with the compliance with the compliance with the criteria. criteria. You don't need to You don't need to do anything do anything do anything special to make the session special to make the session special to make the session last longer than six last longer than six last longer than six hours. She just hours. She just hours. She just has to keep has to keep has to keep working. working. working. One brief One brief One brief clarification. So, if clarification. So, if clarification. So, if I run the agent for I run the agent for I run the agent for many hours and want many hours and want many hours and want to understand the status of the to understand the status of the to understand the status of the checkpoints, is the checkpoints, is the checkpoints, is the best way best way best way to do that through the to do that through the to do that through the session tab? session tab? session tab? Well, yes.

Well, yes. Well, yes. This is the flow of session events. This is the flow of session events. This is the flow of session events. Yes, there is an event API in the session section Yes, there is an event API in the session section Yes, there is an event API in the session section . . . Good. Good. Good. You could create a You could create a You could create a special special special tool, tool, tool, for example, " for example, " for example, " status update for a person", and status update for a person", and status update for a person", and Claude would call this tool at certain Claude would call this tool at certain Claude would call this tool at certain intervals when intervals when intervals when it feels there is it feels there is it feels there is progress progress progress and and and give you an give you an give you an updated status. updated status. updated status. Thank you. Thank you very much. Thank you. Thank you very much. Thank you. Thank you very much. Okay, are there any Okay, are there any Okay, are there any last questions? last questions? last questions? Yes. Yes. Yes. Um, following up on the Um, following up on the Um, following up on the previous question previous question previous question about using about using about using this for Claude tag or this for Claude tag or this for Claude tag or similar similar similar functionality. I functionality. I functionality. I have analyzed many have analyzed many have analyzed many agent frameworks agent frameworks . So if you . So if you . So if you get get get a message while a message while a message while the agent is waiting for the agent is waiting for the agent is waiting for the result of a the result of a the result of a custom custom custom tool, tool, tool, like a human in like a human in like a human in a loop, you said there will a loop, you said there will a loop, you said there will be a 400 error. Do be a 400 error. Do be a 400 error. Do you have any suggestions on you have any suggestions on you have any suggestions on how to handle this? how to handle this? Ahem. Ahem. Can I Can I Can I use use use special special special tools for a tools for a tools for a subagent? I forgot. subagent? I forgot. subagent? I forgot. You can You can You can use use use special special special tools for the tools for the tools for the subagent. subagent. subagent. Because you could Because you could Because you could essentially have a subagent essentially have a subagent essentially have a subagent that handles your that handles your that handles your special special special tool, and then tool, and then tool, and then your main your main your main coordinator agent coordinator agent coordinator agent would always be free would always be free would always be free to receive to receive to receive requests.

requests. requests. Is it possible Is it possible to directly return the result of a to directly return the result of a custom custom custom tool for a tool for a subagent tool in the API? subagent tool in the API? Well, the subagent will issue a Well, the subagent will issue a special special tool usage, and then tool usage, and then tool usage, and then you will return it back you will return it back you will return it back there. there. there. you define your you define your you define your main main coordinator agent, and then coordinator agent, and then coordinator agent, and then you can specify that you can specify that you can specify that agent A talks to agent A talks to agent A talks to agents B and C, and then agents B and C, and then agents B and C, and then agent B will, let's say, agent B will, let's say, agent B will, let's say, handle your handle your handle your own tools, and it's own tools, and it's own tools, and it's in the definition of in the definition of in the definition of agent B that you agent B that you agent B that you configure that configure that configure that tool, and that's going to be the tool, and that's going to be the tool, and that's going to be the session thread that you're session thread that you're session thread that you're going to monitor going to monitor going to monitor to get Is to get Is to get Is it a different session thread it a different session thread it a different session thread or the same one? or the same one? or the same one? No, it's part of the No, it's part of the No, it's part of the same main session. same main session. same main session. Technically a different thread, Technically a different thread, Technically a different thread, but you but you but you only interact with the main only interact with the main only interact with the main thread, and we thread, and we delegate it correctly because we know delegate it correctly because we know where it should where it should where it should go go go based on based on based on the IDs. Ahem. the IDs. Ahem. the IDs. Ahem. Of course. Of course. Of course. And then the main And then the main And then the main agent will pass it on to the agent will pass it on to the agent will pass it on to the subordinate subordinate subordinate agent. agent. agent. So the session is still So the session is still So the session is still inactive, although inactive, although inactive, although if the subordinate if the subordinate if the subordinate agent is doing work, agent is doing work, agent is doing work, it is not inactive. it is not inactive. it is not inactive. This This This But if the master But if the master But if the master coordinating agent coordinating agent coordinating agent is inactive, is inactive, is inactive, subordinate subordinate subordinate agents can still agents can still agents can still continue to work. continue to work. continue to work. Yes, you can Yes, you can Yes, you can see it here. see it here. see it here. Okay, that's pretty Okay, that's pretty Okay, that's pretty impressive.

impressive. impressive. This gap here, the This gap here, the This gap here, the main session is main session is main session is inactive. The main inactive. The main inactive. The main thread is inactive thread is inactive thread is inactive while the subordinate while the subordinate while the subordinate agents are performing agents are performing agents are performing work. work. work. And they can wait And they can wait And they can wait for the results of the for the results of the for the results of the tools. tools. tools. That's right. Yes. So, That's right. Yes. So, That's right. Yes. So, if you're trying to if you're trying to if you're trying to do something where you do something where you do something where you want the main want the main want the main agent to always be agent to always be agent to always be active in active in active in communication, I think the communication, I think the coordinator model is a good coordinator model is a good fit for that. fit for that. fit for that. This is great. This is great. This is great. Yes. Yes. Yes. It looks like we're It looks like we're It looks like we're out of time, but we'll be out of time, but we'll be out of time, but we'll be right right right outside if outside if outside if you want to chat with us you want to chat with us you want to chat with us . . . Thank you for inviting Thank you for inviting Thank you for inviting us. us. us. OK. OK. OK. Thank you very much.