Scheduled Agents, Dreaming and Q&A: Claude Managed Agents (Part 3 of 4) — Anthropic

Okay, so let's Okay, so let's see what it see what it see what it looks like by analyzing looks like by analyzing looks like by analyzing Google and Microsoft. Um, Google and Microsoft. Um, Google and Microsoft. Um, ideally, and I ideally, and I ideally, and I hope this hope this hope this works works works live for you, this live for you, this live for you, this master agent master agent master agent will assess the availability of will assess the availability of will assess the availability of information and then information and then information and then provide us with a provide us with a provide us with a special special special tool with tool with tool with which we will which we will which we will create separate create separate create separate sessions for each of sessions for each of sessions for each of these financial these financial these financial assets. So you assets. So you assets. So you will see this will see this will see this batch analysis process. batch analysis process. batch analysis process. It will call it for It will call it for It will call it for two tickers and then two tickers and then two tickers and then trigger execution trigger execution trigger execution for each of them. This is for each of them. This is for each of them. This is how we can manage how we can manage how we can manage our multi-session our multi-session our multi-session system. And each of system. And each of system. And each of these sessions has a these sessions has a these sessions has a multi-agent multi-agent multi-agent structure that works structure that works structure that works together. And each of them together. And each of them together. And each of them also has a definition of also has a definition of also has a definition of outcomes so that outcomes so that self-monitoring occurs self-monitoring occurs using using using memory stores to teach memory stores to teach memory stores to teach each other and so on. So each other and so on. So each other and so on. So let's move on to let's move on to let's move on to the finale: what if you the finale: what if you the finale: what if you want to do a want to do a want to do a regular job? What regular job? What regular job? What if you want to have a if you want to have a if you want to have a list of companies that list of companies that list of companies that interest you and interest you and interest you and want to schedule an analysis of all of those companies every day at 9:00 want to schedule an analysis of all of those companies every day at 9:00 want to schedule an analysis of all of those companies every day at 9:00 AM or on the first of AM or on the first of AM or on the first of every month every month ? This is what deployments are for.

deployments are for. Deployment is when Deployment is when Deployment is when everything is based on the API. everything is based on the API. everything is based on the API. Of course, you can Of course, you can Of course, you can interact with our interact with our interact with our platform through platform through platform through the console. But the main the console. But the main the console. But the main way we way we way we want you to want you to want you to interact with it interact with it is through the API and SDK. is through the API and SDK. is through the API and SDK. This way, you This way, you This way, you can create can create can create your own interfaces and your own interfaces and your own interfaces and workflows. These are all workflows. These are all workflows. These are all just building just building just building blocks that can be blocks that can be blocks that can be combined to combined to combined to create something bigger. create something bigger. You can see You can see here that Claude created the here that Claude created the here that Claude created the deployment. And it is deployment. And it is deployment. And it is right right right here. The deployment here. The deployment here. The deployment accepts an accepts an accepts an agent definition. In this agent definition. In this agent definition. In this case, it's a weekly case, it's a weekly case, it's a weekly memo from the analytics memo from the analytics memo from the analytics department department department using this using this using this particular agent. particular agent. particular agent. It accepts an It accepts an It accepts an execution schedule and an execution schedule and an execution schedule and an initial set of initial set of initial set of messages that messages that messages that it will work with. it will work with. it will work with. You can also You can also You can also transfer resources transfer resources transfer resources such as such as such as memory storage, memory storage, memory storage, GitHub repositories, or files to be GitHub repositories, or files to be used in used in each of them. So that each of them. So that each of them. So that means it means it means it seems to be on the first of seems to be on the first of seems to be on the first of every month every month every month at 1:00 PM or so.

at 1:00 PM or so. This specific session will be launched This specific session will be launched . I . I . I could run it could run it could run it now, that is, now, that is, now, that is, activate it through the activate it through the activate it through the console. It will be console. It will be console. It will be done. But I done. But I done. But I could also create an could also create an could also create an interface for that, interface for that, interface for that, which is what we did here. which is what we did here. which is what we did here. So I can, you know, So I can, you know, So I can, you know, have this whole crowd. have this whole crowd. have this whole crowd. I can view and I can view and I can view and find my find my find my deployments, and deployments, and deployments, and launch one of them launch one of them launch one of them at any time. And at any time. And at any time. And I don't know why they do I don't know why they do I don't know why they do n't work, but I'll n't work, but I'll n't work, but I'll ask Cloud about it ask Cloud about it ask Cloud about it later. And this has already later. And this has already been mentioned several times, but what been mentioned several times, but what about "dreams", what about "dreams", what about "dreams", what about about about self-improvement? self-improvement? self-improvement? So, before I So, before I So, before I get to your get to your get to your questions, which I'm questions, which I'm questions, which I'm sure are many, I'll sure are many, I'll sure are many, I'll show you a preliminary show you a preliminary show you a preliminary version of what we version of what we version of what we have for "dreams have for "dreams ." So, what is a " ." So, what is a " dream"? It dream"? It dream"? It allows you to choose, well, allows you to choose, well, allows you to choose, well, okay. Before okay. Before okay. Before I show you the I show you the I show you the previous option, previous option, previous option, I will say that it I will say that it I will say that it allows you to choose the allows you to choose the allows you to choose the memory storage. In memory storage. In memory storage. In this case, it this case, it this case, it will use the will use the will use the memory storage that memory storage that memory storage that we were using. You we were using. You we were using. You can choose a can choose a can choose a model. You can model. You can model. You can give her your own give her your own give her your own instructions if instructions if instructions if you want, but we'll you want, but we'll you want, but we'll use the use the use the standard request. And standard request. And standard request. And then we'll let then we'll let then we'll let it analyze it analyze it analyze the sessions from the last 30 the sessions from the last 30 the sessions from the last 30 days using days using days using this " this " dream" analysis. So dream" analysis. So dream" analysis. So when I click " when I click " start dreaming", start dreaming", start dreaming", it starts a session it starts a session it starts a session using using using managed agents, managed agents, managed agents, actually. It runs actually. It runs actually. It runs many sub-agents many sub-agents many sub-agents to analyze to analyze to analyze transcripts of each of the transcripts of each of the transcripts of each of the other sessions we other sessions we other sessions we asked it to evaluate, and asked it to evaluate, and asked it to evaluate, and they all they all they all use use use memory storage. And he memory storage. And he memory storage. And he will analyze: did the will analyze: did the memory store encode all the necessary memory store encode all the necessary information? Did information? Did any issues arise during the session that any issues arise during the session that caused difficulties? Can

caused difficulties? Can caused difficulties? Can we we we learn something from this? Has learn something from this? Has learn something from this? Has this happened this happened this happened repeatedly? Can repeatedly? Can repeatedly? Can we we we somehow update somehow update somehow update the instructions? You can the instructions? You can the instructions? You can see here what he see here what he see here what he actually did: he actually did: he actually did: he found that found that found that the skill we created had the skill we created had the skill we created had a few gaps in the way a few gaps in the way a few gaps in the way it was it was it was written, and he written, and he written, and he updated the updated the updated the memory store to memory store to memory store to let us know that even let us know that even let us know that even though there is a skill, we should though there is a skill, we should though there is a skill, we should supplement that knowledge supplement that knowledge supplement that knowledge with what we have learned. with what we have learned. with what we have learned. You can imagine You can imagine You can imagine that we will continue to that we will continue to that we will continue to expand this. Not expand this. Not expand this. Not only improve only improve only improve memory storage, but also memory storage, but also memory storage, but also improve improve improve skills skills skills directly, directly, directly, agent queries, agent queries, agent queries, sub-agent queries, sub-agent queries, sub-agent queries, etc. That's all. It etc. That's all. It etc. That's all. It was a demonstration. was a demonstration. was a demonstration. Sorry about the Wi-Fi. Sorry about the Wi-Fi. Sorry about the Wi-Fi. I hope you'll I hope you'll I hope you'll follow up later follow up later follow up later when you get when you get when you get home. And let's home. And let's home. And let's continue with the continue with the continue with the questions. I... questions. I... questions. I... Yes, we can all Yes, we can all Yes, we can all answer answer answer questions. questions. questions. Dave, we have a Dave, we have a Dave, we have a question here. question here. question here. Yes. Yes. Yes. Thanks for Thanks for Thanks for the demonstration. We did not the demonstration. We did not the demonstration. We did not see any see any see any telemetry telemetry telemetry instrumentation in the instrumentation in the instrumentation in the agent. Is this a standard agent. Is this a standard agent. Is this a standard built-in feature? built-in feature? built-in feature? What do you mean What do you mean What do you mean by telemetry? Because by telemetry? Because by telemetry? Because you have complete you have complete you have complete observability of observability of observability of all the events that all the events that all the events that have occurred, right?

have occurred, right? have occurred, right? If you go into If you go into If you go into the console, you'll see the console, you'll see the console, you'll see it in real time, it in real time, it in real time, but you can also, of course, but you can also, of course, but you can also, of course, get it get it get it through the API, where this through the API, where this through the API, where this transcript view transcript view transcript view is quite convenient, is quite convenient, is quite convenient, but if you go into but if you go into but if you go into debug mode, debug mode, debug mode, there will be all the events there will be all the events there will be all the events that happened during the that happened during the that happened during the session. So you can session. So you can session. So you can see exactly, see exactly, see exactly, for example, when we for example, when we for example, when we started calling started calling started calling the model? When did we the model? When did we the model? When did we finish calling finish calling finish calling the model? So, you the model? So, you the model? So, you can get can get can get latency metrics. Do latency metrics. Do latency metrics. Do you mean you mean you mean something else by something else by something else by telemetry? telemetry? telemetry? Yes, the thing is, Yes, the thing is, Yes, the thing is, usually in the usually in the usually in the software development cycle, software development cycle, software development cycle, you you you define what you define what you define what you want to observe. want to observe. want to observe. How did we actually choose what we How did we actually choose what we How did we actually choose what we wanted to wanted to wanted to watch? watch? watch? Now you get Now you get Now you get all the events and just all the events and just all the events and just filter them on filter them on filter them on your end. If you your end. If you your end. If you think something is think something is think something is missing, please missing, please missing, please contact us and contact us and contact us and we will discuss we will discuss we will discuss adding it. adding it. adding it. Good. Can I Good. Can I Good. Can I add my own add my own observability stack here? Do observability stack here? Do you mean you mean you mean exporting to OpenTelemetry? exporting to OpenTelemetry? exporting to OpenTelemetry? Yes. Yes. Yes. You could, since You could, since You could, since it's an API, you could it's an API, you could create such a create such a connector quite easily. Everyone has connector quite easily. Everyone has connector quite easily. Everyone has different requirements, and we don't different requirements, and we don't different requirements, and we don't want to impose want to impose want to impose our own opinion, but these are all our own opinion, but these are all our own opinion, but these are all just API calls. just API calls. just API calls. Mhm. Thank you. Do Mhm. Thank you. Do Mhm. Thank you. Do you have a question?

you have a question? you have a question? I had a question I had a question . So, how long . So, how long . So, how long can these can these can these agents work? I know you agents work? I know you agents work? I know you mentioned mentioned mentioned streaming, but streaming, but streaming, but can you run them can you run them can you run them indefinitely, or do you indefinitely, or do you indefinitely, or do you see the see the see the persistent data storage persistent data storage growing all the time and at growing all the time and at some point it some point it some point it needs to be needs to be needs to be compressed offline or compressed offline or compressed offline or something like that? something like that? something like that? A- A- A- We do We do We do the compression for you, the compression for you, the compression for you, so you can so you can so you can run them run them run them for many for many for many days. We've seen 10- days. We've seen 10- hour sessions in hour sessions in hour sessions in production from some of production from some of production from some of our users. our users. our users. You can run You can run You can run these agents for at least a year, these agents for at least a year, these agents for at least a year, or theoretically or theoretically or theoretically indefinitely. indefinitely. indefinitely. Yes, yes. Are Yes, yes. Are Yes, yes. Are you talking about the you talking about the you talking about the agent's work agent's work agent's work over the course of a year or over the course of a year or over the course of a year or one specific session one specific session one specific session that lasts a year? that lasts a year? that lasts a year? No, basically, if I No, basically, if I No, basically, if I need to do need to do need to do the task every day, the task every day, the task every day, can I just can I just can I just leave it leave it leave it running for 6 running for 6 running for 6 months? months? months? Yes, you could. Yes, you could. Yes, you could. Without overflowing its Without overflowing its Without overflowing its persistent persistent persistent data storage. data storage. data storage. That's right. That's right. That's right. And is there any And is there any orchestration tool where I could orchestration tool where I could have three agents have three agents , agent A does something , agent A does something , agent A does something and delegates it to agents B and delegates it to agents B and delegates it to agents B and C? Do you have that and C? Do you have that and C? Do you have that option, or would option, or would option, or would you currently you currently you currently create create create subagents and tell subagents and tell subagents and tell Claude what exactly you want to Claude what exactly you want to Claude what exactly you want to get? We noticed get? We noticed that people have a desire that people have a desire that people have a desire to limit Claude's actions and to limit Claude's actions and to limit Claude's actions and the way in which he the way in which he the way in which he does it. As Claude does it. As Claude does it. As Claude gets smarter, we are gets smarter, we are gets smarter, we are constantly removing constantly removing constantly removing these restrictions and these restrictions and these restrictions and allowing Claude to act allowing Claude to act allowing Claude to act independently. Eh, independently. Eh, independently. Eh, let him let him let him do what he do what he do what he prefers. So, right prefers. So, right prefers. So, right now you think now you think now you think three subagents is the three subagents is the three subagents is the best topology, best topology, best topology, but if we just but if we just but if we just let Claude let Claude let Claude figure it out for himself, the figure it out for himself, the figure it out for himself, the result will be result will be result will be better.

better. better. But is there some kind of But is there some kind of But is there some kind of topological or topological or topological or graph graph graph workflow in the interface workflow in the interface workflow in the interface where I could have… where I could have… where I could have… Not yet. Not yet. Not yet. Not yet. Not yet. Not yet. Not yet, but maybe Not yet, but maybe Not yet, but maybe soon. Yes. E. You soon. Yes. E. You soon. Yes. E. You could probably just could probably just could probably just speak louder. speak louder. speak louder. Yes. Sorry. Sorry. Yes. Sorry. Sorry. Yes. Sorry. Sorry. We will get back to you We will get back to you . . . I apologize for that. I apologize for that. I apologize for that. Hey, we're here, to Hey, we're here, to Hey, we're here, to your right. Yes. So, your right. Yes. So, your right. Yes. So, the question. Looks the question. Looks the question. Looks great. Many great. Many great. Many integration points. I'm integration points. I'm integration points. I'm wondering, you know, for a wondering, you know, for a wondering, you know, for a complex organization complex organization that's potentially that's potentially that's potentially managing a lot of managing a lot of managing a lot of agents, I think they agents, I think they agents, I think they need some kind of need some kind of need some kind of separation, separation, separation, access control (RBAC) or something access control (RBAC) or something access control (RBAC) or something like that. How is this... What is like that. How is this... What is like that. How is this... What is probably the probably the probably the recommended recommended recommended approach for a complex approach for a complex approach for a complex organization that has organization that has organization that has different agents here different agents here different agents here to to to manage effectively? manage effectively? We…we have We…we have workspaces. This is our workspaces. This is our workspaces. This is our basic tool basic tool basic tool today, where you today, where you today, where you can put can put can put different clients in different clients in different clients in different different different workspaces, workspaces, workspaces, segment them segment them segment them that way, and that way, and that way, and give give give specific specific specific users access to those users access to those users access to those spaces. But we spaces. But we spaces. But we know there know there know there are still are still are still gaps in our offering, and that's something gaps in our offering, and that's something gaps in our offering, and that's something we'll have to deal with...

we'll have to deal with... we'll have to deal with... RBAC is in our RBAC is in our RBAC is in our development plan. So we development plan. So we development plan. So we really want really want really want to provide more to provide more to provide more granular granular granular controls. Now controls. Now controls. Now workspaces are a workspaces are a workspaces are a simple way simple way simple way to segregate your to segregate your to segregate your agents, where a group of agents, where a group of agents, where a group of people have access to people have access to people have access to workspace A and workspace A and workspace A and can use can use can use all the agents in all the agents in all the agents in it. You can it. You can it. You can create create create Workspace B with a different Workspace B with a different Workspace B with a different set of users set of users who may have different who may have different who may have different access levels. access levels. access levels. So, hello. Wonderful So, hello. Wonderful So, hello. Wonderful presentation. Very presentation. Very presentation. Very impressed. Two impressed. Two impressed. Two questions. The first is questions. The first is deployment. My deployment. My deployment. My understanding is that our understanding is that our understanding is that our deployment is like a deployment is like a deployment is like a snapshot of all agents snapshot of all agents snapshot of all agents at a certain point in time at a certain point in time . No? . No? . No? Deployment is Deployment is Deployment is essentially a way to essentially a way to essentially a way to launch sessions for a launch sessions for a launch sessions for a specific agent specific agent specific agent on a schedule. You on a schedule. You on a schedule. You can say, "Oh, can say, "Oh, can say, "Oh, research agent, research agent, research agent, research Nvidia once a research Nvidia once a research Nvidia once a quarter." And he will quarter." And he will quarter." And he will create a session create a session create a session every quarter to every quarter to every quarter to conduct research conduct research conduct research and prepare a report for you and prepare a report for you and prepare a report for you . . . Then the memory, which is Then the memory, which is Then the memory, which is the state, is preserved the state, is preserved the state, is preserved between all sessions? between all sessions? between all sessions? That's right. That's right. That's right. You attach You attach You attach memory storage to a memory storage to a memory storage to a session. So you session. So you session. So you decide which storage decide which storage decide which storage to attach to which session to attach to which session to attach to which session . So, . So, . So, storage is an storage is an storage is an independent entity independent entity independent entity that you just that you just that you just plug in when you plug in when you plug in when you need it.

need it. need it. Right. And the last Right. And the last Right. And the last question is about "harnesses" question is about "harnesses" . First, what does First, what does First, what does "harnesses" mean to you, because "harnesses" mean to you, because "harnesses" mean to you, because everyone has their own everyone has their own everyone has their own definition. And the second definition. And the second is confidence. So is confidence. So is confidence. So when I change something here when I change something here when I change something here , how do I know it's , how do I know it's , how do I know it's not... not... not... M-hmm. M-hmm. M-hmm. Yes, as you would expect. Yes, as you would expect. Yes, as you would expect. That's right, my own That's right, my own That's right, my own confidence as a confidence as a confidence as a developer. developer. developer. Uh, yes, you're Uh, yes, you're Uh, yes, you're right. Everyone has their own right. Everyone has their own right. Everyone has their own definition of a definition of a harness. Uh, we, what, what... harness. Uh, we, what, what... harness. Uh, we, what, what... When we at CMA talk When we at CMA talk When we at CMA talk about a framework, we about a framework, we about a framework, we mostly mean the mostly mean the mostly mean the execution of a execution of a execution of a tool cycle. tool cycle. tool cycle. So, managing So, managing So, managing context, context, context, managing managing managing session state, and session state, and session state, and ensuring that the ensuring that the ensuring that the agent can agent can agent can do the work it do the work it do the work it wants to do. When wants to do. When wants to do. When most most most developers talk developers talk developers talk about frameworks, they about frameworks, they about frameworks, they really really really mean...Oh. What they mean...Oh. What they mean...Oh. What they actually mean is actually mean is actually mean is the ability the ability the ability to coordinate multiple to coordinate multiple to coordinate multiple agents and sessions together agents and sessions together agents and sessions together as a unit of work to as a unit of work to as a unit of work to accomplish some accomplish some accomplish some larger task in a larger task in a larger task in a given topology. We given topology. We given topology. We provide you with an provide you with an provide you with an abstraction of a single abstraction of a single abstraction of a single session as a unit of session as a unit of session as a unit of work so that you can work so that you can work so that you can go and build go and build go and build your framework your framework your framework using a session using a session using a session as one of the primitives as one of the primitives . Uh-huh.

. Uh-huh. . Uh-huh. And regarding your second And regarding your second And regarding your second question about how question about how question about how to find out how to find out how to find out how my agent is doing? Our my agent is doing? Our my agent is doing? Our best best best recommendation is recommendation is to run assessments for to run assessments for to run assessments for your agent on your agent on your agent on different types of tasks different types of tasks different types of tasks you want them to you want them to you want them to perform and see perform and see how they perform on them how they perform on them how they perform on them . " . " Results" is a really Results" is a really Results" is a really good way good way good way we we we direct an agent direct an agent direct an agent to get a to get a to get a specific result specific result , where you can , where you can , where you can define the exact define the exact define the exact criteria: "Hey, this is criteria: "Hey, this is criteria: "Hey, this is exactly the result exactly the result exactly the result I want." You I want." You I want." You can set a can set a can set a maximum maximum maximum number of iterations and the number of iterations and the number of iterations and the agent will continue to agent will continue to agent will continue to run until it run until it run until it reaches that reaches that reaches that result. Yes, the result. Yes, the result. Yes, the verification agent verification agent Gabe was talking about. Gabe was talking about. Greetings. Therefore. after Greetings. Therefore. after Greetings. Therefore. after Class. Hey, thanks for the Class. Hey, thanks for the Class. Hey, thanks for the demo. I demo. I demo. I liked it. Uh, my liked it. Uh, my liked it. Uh, my question is essentially about question is essentially about question is essentially about how how how to choose between to choose between to choose between skills and skills and skills and agents. And I wanted agents. And I wanted agents. And I wanted to ask, what would you to ask, what would you to ask, what would you say is the say is the say is the fundamental fundamental fundamental difference? Is there an difference? Is there an difference? Is there an advantage in the framework advantage in the framework advantage in the framework around the agent that around the agent that around the agent that makes it produce makes it produce makes it produce better results at better results at better results at scale? One of the scale? One of the scale? One of the reasons I reasons I reasons I ask is because ask is because ask is because I'm afraid of I'm afraid of I'm afraid of constantly investing constantly investing constantly investing in new kinds of frameworks in new kinds of frameworks in new kinds of frameworks and tools and tools and tools around models, and it around models, and it around models, and it can be tiring— can be tiring— constantly trying constantly trying constantly trying to do something new. So to do something new. So to do something new. So yeah, I just want to yeah, I just want to yeah, I just want to hear what you think: is there a hear what you think: is there a hear what you think: is there a big difference and will big difference and will big difference and will it change it change it change anytime soon?

anytime soon? Yes, no. Thick Yes, no. Thick frame, thin frame frame, thin frame , thick skills—this is a , thick skills—this is a , thick skills—this is a question that is constantly question that is constantly question that is constantly evolving. I think evolving. I think evolving. I think we see that as we see that as we see that as Claude gets Claude gets Claude gets smarter, the boundaries and the smarter, the boundaries and the smarter, the boundaries and the help we help we help we want to give Claude want to give Claude want to give Claude start to melt away and start to melt away and start to melt away and disappear. So what disappear. So what disappear. So what you need you need you need today may today may today may not be needed tomorrow. The not be needed tomorrow. The not be needed tomorrow. The boring answer is to boring answer is to just run just run just run the assessment and the assessment and the assessment and check these things. check these things. check these things. It sounds It sounds It sounds exhausting, because exhausting, because exhausting, because every time a every time a every time a new model comes out, I new model comes out, I new model comes out, I have to check all of this have to check all of this have to check all of this . We see . We see that skills are a that skills are a that skills are a good abstraction for good abstraction for good abstraction for today. This gives today. This gives today. This gives the model enough the model enough the model enough information to information to information to perform its tasks. perform its tasks. But I assume that over But I assume that over time, things will change and time, things will change and time, things will change and become simpler when become simpler when become simpler when the model can do the model can do the model can do everything on its own without everything on its own without everything on its own without additional requests additional requests additional requests or instructions, so the or instructions, so the or instructions, so the multi-agent multi-agent multi-agent approach seems approach seems approach seems more useful, but more useful, but more useful, but the results may the results may the results may depend on your depend on your depend on your task. task.

Congratulations. Congratulations. Greetings. Congratulations. Greetings. Congratulations. Greetings. Congratulations. My question My question My question is related to the is related to the is related to the previous two, especially previous two, especially previous two, especially regarding evaluation. I am regarding evaluation. I am regarding evaluation. I am very interested in " very interested in " dreaming", dreaming", dreaming", but how will it affect the but how will it affect the but how will it affect the final result? This is where final result? This is where assessments come in handy. assessments come in handy. But you can have But you can have But you can have multiple iterations multiple iterations multiple iterations of dreams that can of dreams that can of dreams that can lead in different lead in different lead in different directions. Is there directions. Is there directions. Is there any plan or do any plan or do any plan or do you support you support you support automated automated automated dreaming to dreaming to dreaming to run an evaluation run an evaluation run an evaluation and see what and see what and see what different different different directions lead to? Which one directions lead to? Which one directions lead to? Which one to choose depending on to choose depending on to choose depending on what is happening what is happening ? ? ? Yes, "dreaming" Yes, "dreaming" Yes, "dreaming" works like this: you can works like this: you can works like this: you can choose a period of time choose a period of time choose a period of time during which the agent during which the agent during which the agent should "dream." You should "dream." You should "dream." You can say, "I can say, "I can say, "I want the agent want the agent want the agent to analyze the sessions to analyze the sessions to analyze the sessions from the last 30 days." from the last 30 days." from the last 30 days." It will review all It will review all It will review all the transcripts, the transcripts, the transcripts, analyze the tasks analyze the tasks you attempted to you attempted to you attempted to complete, and create a complete, and create a complete, and create a new memory store new memory store new memory store more tailored more tailored more tailored to your goals. We to your goals. We to your goals. We routinely run routinely run routinely run evaluations on our evaluations on our evaluations on our end to end to end to ensure that " ensure that " dreaming" delivers dreaming" delivers dreaming" delivers better results, and better results, and better results, and we see customers we see customers we see customers achieve significant achieve significant performance improvements performance improvements using this using this using this approach and the new approach and the new approach and the new memory storage. But again memory storage. But again memory storage. But again , it all comes down , it all comes down , it all comes down to evaluation. This is how to evaluation. This is how to evaluation. This is how you will know if you will know if you will know if the system is working as the system is working as the system is working as you expect. Mhm.

you expect. Mhm. you expect. Mhm. Yes, I think it would be Yes, I think it would be Yes, I think it would be good to have our own good to have our own good to have our own set of assessments. Ahem. set of assessments. Ahem. set of assessments. Ahem. Another thing is that " Another thing is that " dreaming" is currently in the dreaming" is currently in the research research preview stage preview stage preview stage . These are early . These are early . These are early stages, a big stages, a big stages, a big experiment. Overall, experiment. Overall, experiment. Overall, we see improvement, we see improvement, we see improvement, but we heard you. but we heard you. but we heard you. Obviously, how do I Obviously, how do I Obviously, how do I get confidence that get confidence that get confidence that this is... this is... this is... Yes, it's related to Yes, it's related to Yes, it's related to the evaluation of " the evaluation of " dreams," we can dreams," we can dreams," we can trace the different trace the different trace the different paths that paths that paths that the model has taken and decide, "Oh the model has taken and decide, "Oh , this makes more sense , this makes more sense ." Because even if ." Because even if ." Because even if nine sessions could nine sessions could nine sessions could indicate that this is indicate that this is indicate that this is wrong. One wrong. One wrong. One says, hey, that's good. says, hey, that's good. says, hey, that's good. So if you So if you So if you rely solely on rely solely on rely solely on these "dream" sessions, these "dream" sessions, these "dream" sessions, I feel like we I feel like we I feel like we might not get might not get might not get all the insights without all the insights without all the insights without evaluation. evaluation. evaluation. If I tell you to If I tell you to If I tell you to just trust just trust just trust me, is that enough me, is that enough ? ? Thank you. Yes. Yes. OK. Uh, OK. Uh, OK. Uh, thank you very much. Hmm, that's very thank you very much. Hmm, that's very thank you very much. Hmm, that's very impressive. Those of us who have impressive. Those of us who have impressive. Those of us who have been working with Cloud Code for been working with Cloud Code for been working with Cloud Code for a year or more a year or more a year or more have been doing similar things, have been doing similar things, have been doing similar things, but it's nice to see it but it's nice to see it but it's nice to see it in a truly mature in a truly mature in a truly mature form. Um, I have a form. Um, I have a form. Um, I have a question: to what extent question: to what extent question: to what extent was this built was this built was this built solely on Cloud solely on Cloud solely on Cloud Code? I mean, if someone Code? I mean, if someone Code? I mean, if someone was, you know, ambitious, was, you know, ambitious, was, you know, ambitious, could they could they could they create something create something create something like that? And, um, I wanted like that? And, um, I wanted like that? And, um, I wanted to ask you, uh, to ask you, uh, to ask you, uh, encourage you encourage you encourage you to keep things to keep things to keep things open. Some of open. Some of open. Some of us are already collecting data from us are already collecting data from us are already collecting data from session JSON files. And session JSON files. And session JSON files. And starts starts starts experimenting with experimenting with experimenting with things like heparag things like heparag things like heparag etc. for

etc. for etc. for memory management and creating a memory management and creating a memory management and creating a memory pool. So, uh, memory pool. So, uh, memory pool. So, uh, congratulations. Thank you. congratulations. Thank you. congratulations. Thank you. Please keep this Please keep this Please keep this open, and are open, and are open, and are you doing anything here you doing anything here you doing anything here that isn't just that isn't just that isn't just based on Cloud Code? based on Cloud Code? Thank you. Thank you. We took the We took the We took the Cloud Code toolkit Cloud Code toolkit Cloud Code toolkit as a source of inspiration. as a source of inspiration. as a source of inspiration. We found it to be We found it to be We found it to be very well very well very well adapted for adapted for programming tasks. Um, and programming tasks. Um, and it has varying levels of it has varying levels of it has varying levels of flexibility for other flexibility for other flexibility for other tasks. We build tasks. We build tasks. We build APIs where we want you to APIs where we want you to APIs where we want you to bring your bring your bring your context, information, context, information, context, information, and capabilities and and capabilities and and capabilities and build on top of build on top of build on top of it, whereas Cloud Code it, whereas Cloud Code it, whereas Cloud Code tries to give you a tries to give you a tries to give you a finished product. So finished product. So finished product. So we have somewhat different, we have somewhat different, we have somewhat different, you know, value you know, value you know, value propositions. Um, and so propositions. Um, and so propositions. Um, and so it's not just Cloud Code in a it's not just Cloud Code in a it's not just Cloud Code in a box. Although, you know, box. Although, you know, box. Although, you know, ideally it works just as well ideally it works just as well ideally it works just as well , if , if , if not better, in all types of not better, in all types of not better, in all types of tasks. Um, so, not exactly tasks. Um, so, not exactly tasks. Um, so, not exactly , but it's , but it's , but it's definitely inspired and definitely inspired and definitely inspired and motivated by some of the motivated by some of the motivated by some of the improvements we've improvements we've improvements we've seen in Cloud Code.

seen in Cloud Code. And Cloud Code is a And Cloud Code is a terminal terminal terminal tool, and we tool, and we tool, and we obviously want to be an obviously want to be an obviously want to be an API product and API product and API product and infrastructure that infrastructure that infrastructure that will scale will scale will scale and be reliable. So, and be reliable. So, and be reliable. So, there were some challenges that there were some challenges that there were some challenges that I told you about I told you about I told you about at the beginning of at the beginning of at the beginning of the presentation the presentation the presentation that we that we that we worked hard on to worked hard on to worked hard on to provide you with a provide you with a provide you with a truly reliable truly reliable truly reliable product to product to product to use. use. Yes. So, for AWS Bedrock, is there a Yes. So, for AWS Bedrock, is there a console there console there console there similar to the cloud similar to the cloud similar to the cloud console that you have console that you have console that you have here? This is a pretty straightforward here? This is a pretty straightforward here? This is a pretty straightforward question, yes, I question, yes, I question, yes, I just want to control just want to control just want to control everything and see everything. everything and see everything. everything and see everything. This is not available in Bedrock, This is not available in Bedrock, This is not available in Bedrock, but is available through the but is available through the but is available through the cloud platform for cloud platform for cloud platform for AWS. So, there is an AWS AWS. So, there is an AWS version of the console that version of the console that version of the console that you can you can you can use for use for use for this. this. Yes. Yes. Uh, first of all, uh , are there any , are there any , are there any restrictions on the restrictions on the restrictions on the model that can be model that can be model that can be used, used, used, for example, models not for example, models not for example, models not from Anthropic? Uh from Anthropic? Uh from Anthropic? Uh -huh. -huh. -huh. Can I run Can I run Can I run cloud code on cloud code on cloud code on non-Anthropic models? non-Anthropic models?

non-Anthropic models? For now, we are you. For now, we are you. For now, we are you. Yes, only Yes, only Yes, only cloud models for now. Uh cloud models for now. Uh cloud models for now. Uh , okay. So, the second , okay. So, the second , okay. So, the second question is: what about question is: what about end- end- user usage? Do user usage? Do user usage? Do they have to have they have to have they have to have an account in the an account in the an account in the console to do this, or console to do this, or console to do this, or can I create my can I create my can I create my own interface own interface own interface for them to for them to for them to use? use? use? So, the cloud console So, the cloud console So, the cloud console helps you helps you helps you create your API key. create your API key. create your API key. This is how you This is how you This is how you access our APIs. access our APIs. access our APIs. So, you don't pay So, you don't pay So, you don't pay separately for the cloud separately for the cloud separately for the cloud console. Registration is console. Registration is console. Registration is free. You free. You free. You create an API key and create an API key and create an API key and then pay based on then pay based on token usage. token usage. So for a case So for a case So for a case like SaaS where like SaaS where like SaaS where we might have multiple we might have multiple we might have multiple tenants, right? tenants, right? So to provide this So to provide this agent, what model have you agent, what model have you agent, what model have you already developed that we already developed that we already developed that we can follow, can follow, can follow, for example, should for example, should for example, should each company or each company or each company or tenant have a separate tenant have a separate tenant have a separate space, or even a space, or even a space, or even a separate module for separate module for separate module for each tenant? So each tenant? So , how are we actually , how are we actually , how are we actually going to going to going to organize this from organize this from organize this from that perspective?

that perspective? that perspective? We have seen that We have seen that We have seen that the best model the best model the best model for multi-tenancy for multi-tenancy for multi-tenancy is not to "bring your own is not to "bring your own is not to "bring your own key" as you key" as you key" as you lose the ability lose the ability lose the ability to observe because to observe because to observe because it works through it works through your your your end end end user's API key, not your user's API key, not your user's API key, not your own. So we own. So we own. So we encourage you to encourage you to encourage you to model things model things model things using using using workspaces as a workspaces as a workspaces as a tenancy primitive. tenancy primitive. tenancy primitive. Uh, and we're Uh, and we're Uh, and we're refining this to make it refining this to make it refining this to make it easier and easier and easier and easier to build easier to build easier to build solutions on this for solutions on this for solutions on this for multi-tenant multi-tenant multi-tenant applications like. applications like. applications like. Yes, I think now Yes, I think now Yes, I think now you can either you can either you can either put all put all put all your customer data in one your customer data in one your customer data in one workspace and workspace and workspace and then maintain then maintain then maintain the mapping on the mapping on the mapping on your side of which your side of which your side of which agent corresponds to agent corresponds to agent corresponds to which of your which of your which of your customers. customers. customers. Alternatively, as Alternatively, as Alternatively, as Gabe said, you can Gabe said, you can Gabe said, you can create a create a create a workspace for each workspace for each workspace for each client, but that can be client, but that can be client, but that can be cumbersome because you would cumbersome because you would cumbersome because you would then have to then have to then have to recreate your recreate your recreate your agent. agent. agent. So have you seen So have you seen So have you seen examples where people examples where people examples where people have added an additional have added an additional have added an additional layer of security or RBAC layer of security or RBAC layer of security or RBAC on top of this on top of this on top of this workspace, because workspace, because workspace, because to me it's a little...well, to me it's a little...well, to me it's a little...well, you get the idea. you get the idea. Yes, this is probably a Yes, this is probably a good way to good way to good way to do it right now, do it right now, do it right now, since we don't have since we don't have since we don't have any any any RBAC controls at our RBAC controls at our RBAC controls at our level other than the level other than the level other than the workspace. Since workspace. Since workspace. Since everything within your everything within your everything within your workspace is workspace is workspace is accessible to everyone in it accessible to everyone in it , but we're working on , but we're working on , but we're working on improving RBAC and the improving RBAC and the improving RBAC and the overall experience, overall experience, overall experience, if you're working with if you're working with if you're working with cloud agents cloud agents cloud agents building solutions building solutions building solutions for customers—how can we for customers—how can we for customers—how can we make it easier?

make it easier? make it easier? So can you So can you So can you show how to show how to show how to scale this if there scale this if there scale this if there are many, how to are many, how to are many, how to say, tenants say, tenants say, tenants working with this? working with this? working with this? Yes. Yes. Scaling parameters, like Scaling parameters, like say, minimum say, minimum say, minimum pod or any other pod or any other pod or any other parameters parameters parameters responsible for this. responsible for this. I mean, you I mean, you define it yourself, you can define it yourself, you can define it yourself, you can create as create as create as many sessions as you want. There are many sessions as you want. There are no restrictions. We have some no restrictions. We have some limits on limits on limits on query frequency, of course, but query frequency, of course, but query frequency, of course, but there are no there are no there are no scaling options so you scaling options so you scaling options so you can say, "I can say, "I can say, "I need to run need to run need to run 10,000 pods." 10,000 pods." 10,000 pods." So there's So there's So there's nothing like nothing like nothing like HPA in Kubernetes right now where you can HPA in Kubernetes right now where you can HPA in Kubernetes right now where you can configure configure scaling parameters like scaling parameters like timeout or something? timeout or something? timeout or something? You don't need it. You don't need it. You don't need it. This is not necessary, because we This is not necessary, because we This is not necessary, because we take all the take all the take all the management upon ourselves. management upon ourselves. management upon ourselves. You just make You just make You just make API calls, and that's it. API calls, and that's it. API calls, and that's it. Of course. OK. Of course. OK. Of course. OK. Of course. Thanks for Of course. Thanks for Of course. Thanks for the reply. By the way, the reply. By the way, the reply. By the way, the link to the link to the link to the offer doesn't work for me the offer doesn't work for me the offer doesn't work for me . Not sure . Not sure . Not sure if it's just me, if it's just me, if it's just me, but yes. Thank you for but yes. Thank you for but yes. Thank you for sharing. sharing. sharing. Thank you. Thank you. Thank you. Yes, continue.

Yes, continue. Yes, continue. Congratulations. First of all, Congratulations. First of all, Congratulations. First of all, thank you very much. I get thank you very much. I get thank you very much. I get great great great pleasure from pleasure from pleasure from everything you everything you everything you produce. And I'm produce. And I'm produce. And I'm really curious about how you really curious about how you really curious about how you use the UI. You use the UI. You use the UI. You run the server, you run the server, you run the server, you distribute it, distribute it, distribute it, but how do we deploy but how do we deploy but how do we deploy it internally, or, it internally, or, it internally, or, more to my interest, more to my interest, more to my interest, to the customer base? to the customer base? If I want to create, If I want to create, for example, an astrologer, for example, an astrologer, for example, an astrologer, MGM. MGM. MGM. This person will have to This person will have to This person will have to enter enter enter sensitive data, sensitive data, sensitive data, so it would be great to so it would be great to so it would be great to be able to be able to be able to implement this in the implement this in the implement this in the script. script. script. have a level of protection. have a level of protection. have a level of protection. Yes. Yes. Yes. So, there are certain So, there are certain So, there are certain procedural points. procedural points. procedural points. Take, for example, a Take, for example, a Take, for example, a star map. I'm not star map. I'm not star map. I'm not building this, but it's a building this, but it's a building this, but it's a good example. good example. good example. For example, I can For example, I can For example, I can instruct a subagent instruct a subagent instruct a subagent to do to do to do an interpretation, which an interpretation, which an interpretation, which might take about might take about might take about 90 seconds. This can 90 seconds. This can 90 seconds. This can create certain create certain create certain assets, etc. When assets, etc. When assets, etc. When the result is achieved, the result is achieved, the result is achieved, the system can the system can the system can shut down. So this shut down. So this shut down. So this allows me to allows me to allows me to do a really clean do a really clean do a really clean implementation where I implementation where I implementation where I only need to only need to only need to disable. disable. disable. Yes. Yes. Yes. And then everything can And then everything can And then everything can start from this start from this start from this moment. moment. moment. Yes, that's right. Yes, that's right. Yes, that's right. Several things, like Several things, like Several things, like MCP channels, that MCP channels, that MCP channels, that point to this, point to this, point to this, for example, through for example, through for example, through fake chat. The way fake chat. The way fake chat. The way for you is to simply for you is to simply for you is to simply use Claude use Claude use Claude and have a channel to and have a channel to and have a channel to communicate with him through the communicate with him through the communicate with him through the interface. But I see interface. But I see interface. But I see a lot of cases where a lot of cases where a lot of cases where you just run a you just run a you just run a server inside a server inside a server inside a project that's easy project that's easy project that's easy to distribute to distribute to distribute internally, but how do you internally, but how do you internally, but how do you deploy that at deploy that at deploy that at scale?

scale? Um, I mean, this is Um, I mean, this is already being deployed already being deployed already being deployed today at an today at an today at an industrial industrial industrial scale, for example, scale, for example, scale, for example, Notion's external agents Notion's external agents Notion's external agents are built on this. are built on this. are built on this. So that's what exists So that's what exists So that's what exists today, and you can today, and you can today, and you can use use use cloud-managed cloud-managed cloud-managed agents through Notion and agents through Notion and agents through Notion and Atlassian to integrate Atlassian to integrate Atlassian to integrate with Jira, for example. with Jira, for example. with Jira, for example. There are ways to There are ways to There are ways to do this. I think do this. I think do this. I think if you have if you have if you have specific questions, specific questions, specific questions, we can discuss. we can discuss. we can discuss. Yes, we can Yes, we can Yes, we can talk after talk after talk after that, but usually that, but usually that, but usually you create your you create your you create your product, your product, your product, your interface, which you interface, which you interface, which you distribute to distribute to distribute to customers, and then you customers, and then you customers, and then you contact us for contact us for contact us for your agent. your agent. your agent. The phrase "all" is yes, all The phrase "all" is yes, all The phrase "all" is yes, all via API. via API. via API. Everything is through the API. So, you will Everything is through the API. So, you will Everything is through the API. So, you will manage all the manage all the manage all the API calls and API calls and API calls and build your build your build your product and interface product and interface product and interface around that. Yes. around that. Yes. Congratulations. Congratulations. Congratulations. Congratulations. First of all Congratulations. Congratulations. First of all , thank you for the great , thank you for the great , thank you for the great seminar and demo. My seminar and demo. My seminar and demo. My question is not entirely question is not entirely question is not entirely about the about the about the agent himself. I just agent himself. I just agent himself. I just saw that saw that saw that Sonnet 3.5 was just released. Sonnet 3.5 was just released. Sonnet 3.5 was just released. Yes. Yes. Yes. I was just wondering I was just wondering I was just wondering if you could if you could if you could share something with share something with share something with us.

us. us. Yes. Yes. Yes. This is a great model. This is a great model. This is a great model. Yes, try it. Yes, try it. It is supported in It is supported in cloud-managed cloud-managed cloud-managed agents. Try it. agents. Try it. agents. Try it. You can You can You can use it use it use it today. today. today. Greetings. If we have Greetings. If we have Greetings. If we have business partners business partners business partners using, using, using, for example, Claude Code, or for example, Claude Code, or for example, Claude Code, or some more technically some more technically some more technically savvy people savvy people savvy people using Claude using Claude using Claude Code to build Code to build Code to build point solutions, what point solutions, what point solutions, what level of complexity is level of complexity is level of complexity is needed to start needed to start needed to start using using using managed agents? Because managed agents? Because managed agents? Because it looks it looks it looks more powerful, but a cloud more powerful, but a cloud more powerful, but a cloud platform can be platform can be platform can be intimidating to people who are not intimidating to people who are not intimidating to people who are not engineers. engineers. engineers. Yeah, I think the quick Yeah, I think the quick Yeah, I think the quick start in the console—that's start in the console—that's start in the console—that's probably the part of probably the part of probably the part of my demo where my demo where my demo where you just tell Cloud you just tell Cloud you just tell Cloud what kind of agent you what kind of agent you what kind of agent you 're trying to create 're trying to create , which should make the , which should make the , which should make the process process process simple enough for even simple enough for even simple enough for even non-technical non-technical non-technical users to users to users to create create create their first agent. their first agent. their first agent. But obviously, this is an API But obviously, this is an API product, and it product, and it product, and it will naturally be a little will naturally be a little will naturally be a little more complex, but more complex, but more complex, but they are already used to Cloud they are already used to Cloud they are already used to Cloud Code. We have a Code. We have a Code. We have a guided agent skill in guided agent skill in guided agent skill in Cloud Code that will essentially Cloud Code that will essentially guide you through guide you through building your building your building your first agent in natural language, and I would first agent in natural language, and I would first agent in natural language, and I would recommend trying recommend trying recommend trying that. I think the that. I think the that. I think the biggest difference is biggest difference is biggest difference is that in Cloud Code you that in Cloud Code you that in Cloud Code you create a create a create a personal agent personal agent . Here, you are most likely . Here, you are most likely . Here, you are most likely trying to trying to trying to create an agent that create an agent that create an agent that will be available to your will be available to your will be available to your team, your team, your team, your internal internal internal organization, or you organization, or you organization, or you are creating a product are creating a product are creating a product that you deliver to that you deliver to that you deliver to end customers.

end customers. end customers. But if you're building But if you're building But if you're building personal agents personal agents , Cloud Code remains a , Cloud Code remains a , Cloud Code remains a great tool great tool great tool for that. Are for that. Are for that. Are there any there any there any skills to take skills to take skills to take something that already exists in something that already exists in something that already exists in code and turn code and turn code and turn it into documentation? it into documentation? it into documentation? Mhm. There is a skill for Mhm. There is a skill for Mhm. There is a skill for interacting with our API, interacting with our API, interacting with our API, so Claude will probably be able so Claude will probably be able so Claude will probably be able to handle to handle to handle it well enough. So, if you it well enough. So, if you it well enough. So, if you say, "Here's what I say, "Here's what I say, "Here's what I did in code, and here's did in code, and here's did in code, and here's the skill for calling the the skill for calling the the skill for calling the API, do the translation." I'm API, do the translation." I'm API, do the translation." I'm pretty sure. pretty sure. pretty sure. Yes. Yes. Yes. He will do it for He will do it for He will do it for you. you. Um, my question is Um, my question is about the about the about the corporate corporate corporate sector. Many sector. Many sector. Many enterprises would like to enterprises would like to enterprises would like to run this in- run this in- run this in- house house house because they have because they have because they have sensitive sensitive sensitive data sets that they don't want to data sets that they don't want to data sets that they don't want to take out. take out. take out. Additionally, Additionally, Additionally, they may have already set up they may have already set up they may have already set up their own secure their own secure their own secure proxies to proxies to proxies to Anthropic models. Um, are you, and Anthropic models. Um, are you, and Anthropic models. Um, are you, and another question was another question was another question was about scaling. about scaling. about scaling. So, within an So, within an So, within an enterprise, enterprise, enterprise, you can now allocate you can now allocate you can now allocate computing computing computing resources to resources to resources to scale it scale it scale it your own way. your own way. your own way. Right? So, have Right? So, have Right? So, have you considered an you considered an you considered an on-prem on-prem on-prem solution? solution? This is definitely a request This is definitely a request from a lot of our from a lot of our from a lot of our corporate corporate corporate clients and we are clients and we are clients and we are seriously considering seriously considering seriously considering it right now, but we don't have it right now, but we don't have it right now, but we don't have anything to anything to share yet, but we are share yet, but we are really seriously really seriously really seriously considering it. Yes.

considering it. Yes. considering it. Yes. Sorry. Could Sorry. Could Sorry. Could you show you show you show the last slide with the QR the last slide with the QR code of the seminar? Are code of the seminar? Are you looking for you looking for a repository or a repository or a repository or credits? Repository. credits? Repository. credits? Repository. OK.