Containment Is Lying to You: Measuring AI Agents Like Employees — Druid AI

Good morning. I'm Good morning. I'm Michael, the Michael, the developer community manager at Juji AI, and developer community manager at Juji AI, and this is Dan, our this is Dan, our this is Dan, our co-founder and co-founder and co-founder and chief chief chief product officer. Briefly product officer. Briefly product officer. Briefly about who we are. Juji about who we are. Juji is a is a is a conversational AI startup with conversational AI startup with conversational AI startup with hundreds of customers in hundreds of customers in hundreds of customers in EMEA and the EMEA and the EMEA and the Americas, and we have Americas, and we have Americas, and we have been operating for several been operating for several been operating for several years. Because we years. Because we years. Because we work with many work with many work with many clients who are clients who are clients who are implementing agents implementing agents implementing agents into production, we into production, we into production, we hear the same hear the same hear the same question all the time. How do question all the time. How do question all the time. How do we really we really we really know if know if know if our AI agent is effective our AI agent is effective ? What numbers ? What numbers matter? What matter? What matter? What data should be data should be data should be checked? There checked? There checked? There was no clear answer, was no clear answer, was no clear answer, so we created so we created so we created our own benchmark our own benchmark our own benchmark model and model and model and rating system that rating system that rating system that we use to we use to we use to help clients help clients help clients evaluate their evaluate their evaluate their agents. One agents. One agents. One way we way we way we explain this to our explain this to our explain this to our clients is with a simple clients is with a simple clients is with a simple analogy, well, interns, analogy, well, interns, analogy, well, interns, right? Who doesn't love right? Who doesn't love right? Who doesn't love interns? That is why interns? That is why interns? That is why we will guide you through we will guide you through we will guide you through this session today this session today . So, here's the scenario.

. So, here's the scenario. . So, here's the scenario. Let's take a Let's take a Let's take a business scenario, shall we? business scenario, shall we? You are a manager at a bank's customer service center and bank's customer service center and are hiring two are hiring two are hiring two interns for the summer. interns for the summer. One team, One team, same job same job same job descriptions for both descriptions for both descriptions for both interns. Until August, interns. Until August, interns. Until August, this is what their this is what their this is what their dashboard shows, dashboard shows, dashboard shows, right? Intern A right? Intern A right? Intern A processes 80% of the conversations, processes 80% of the conversations, processes 80% of the conversations, passes 20% on to his passes 20% on to his passes 20% on to his senior bankers. senior bankers. Intern B processes 96 % and passes on four % and passes on four % and passes on four percent, almost percent, almost percent, almost nothing. And the nothing. And the nothing. And the control panel is green. control panel is green. control panel is green. Now you have one Now you have one Now you have one offer of offer of offer of permanent permanent permanent employment. employment. employment. Who will get a Who will get a Who will get a full-time full-time full-time job offer? At job offer? At job offer? At first glance, Intern first glance, Intern first glance, Intern B seems like an B seems like an B seems like an obvious candidate, obvious candidate, obvious candidate, right? Intern B right? Intern B right? Intern B handled almost everything on his handled almost everything on his handled almost everything on his own, without own, without own, without constantly distracting the constantly distracting the constantly distracting the senior bankers senior bankers senior bankers with conversations. But, strangely enough with conversations. But, strangely enough with conversations. But, strangely enough , , , Intern B's offer is the Intern B's offer is the Intern B's offer is the wrong choice. wrong choice.

Here's what actually Here's what actually happened to Intern A happened to Intern A happened to Intern A as an example. A customer as an example. A customer authenticates at 7:00 authenticates at 7:00 AM on the mobile AM on the mobile AM on the mobile app and asks, “ app and asks, “ What do I need to do What do I need to do What do I need to do to to to pay off my mortgage early?” pay off my mortgage early?” The agent cannot The agent cannot answer answer answer math questions, math questions, math questions, get a balance, get a balance, get a balance, perform a calculation, perform a calculation, perform a calculation, and provide a complete and provide a complete and provide a complete answer. This is included answer. This is included answer. This is included in any definition in any definition . But the question is, this is . But the question is, this is . But the question is, this is a signal, the client a signal, the client a signal, the client is rethinking their is rethinking their is rethinking their mortgage, this is mortgage, this is mortgage, this is refinancing, this is refinancing, this is refinancing, this is real profit on the real profit on the real profit on the table. Therefore, the bank writes table. Therefore, the bank writes table. Therefore, the bank writes a rule: respond to a rule: respond to a rule: respond to the client, and then the client, and then the client, and then direct the contact direct the contact direct the contact to a senior banker. to a senior banker. to a senior banker. This planned This planned This planned transfer is part of transfer is part of transfer is part of that 20% of trainee A. that 20% of trainee A. that 20% of trainee A. The metric that The metric that The metric that describes this is the describes this is the describes this is the containment level. containment level. containment level. The percentage of conversations The percentage of conversations The percentage of conversations conducted without conducted without conducted without human intervention is easy human intervention is easy human intervention is easy to measure and is available on to measure and is available on to measure and is available on every dashboard every dashboard every dashboard . This is exactly the . This is exactly the . This is exactly the gap that gap that our clients have consistently encountered, and that is why our clients have consistently encountered, and that is why we have created a we have created a we have created a benchmark model and benchmark model and benchmark model and scoring system, and scoring system, and scoring system, and Dan will walk you through Dan will walk you through Dan will walk you through this framework.

this framework. Thank you. Thank you, Michael. Um, this is a framework that Um, this is a framework that we created because we created because we created because our customers and our customers and our customers and prospects prospects prospects have been asking us, "How have been asking us, "How have been asking us, "How do you know that an agent, do you know that an agent, do you know that an agent, our agent, our agent, our agent, is doing is doing is doing its job? How do we its job? How do we its job? How do we know that an agent is know that an agent is know that an agent is effective?" So we effective?" So we effective?" So we built this built this built this framework on six framework on six framework on six pillars, divided pillars, divided pillars, divided into two groups with into two groups with into two groups with governance that governance that governance that covers all of those pillars covers all of those pillars . The configuration group . The configuration group is what the is what the is what the agent is made of. What agent is made of. What exists before the exists before the exists before the agent is launched. And the runtime is what the agent is launched. And the runtime is what the agent does while it's agent does while it's running. What tasks running. What tasks running. What tasks it creates, what tasks it it creates, what tasks it it creates, what tasks it executes, and what the executes, and what the executes, and what the outcome is. Governance flows through all of outcome is. Governance flows through all of outcome is. Governance flows through all of those groups those groups those groups . From a . From a . From a business perspective, what are the business perspective, what are the business perspective, what are the expected boundaries? And expected boundaries? And expected boundaries? And within governance, we within governance, we within governance, we observe and observe and observe and control control control error tolerance, risk, which is error tolerance, risk, which is error tolerance, risk, which is very important, very important, very important, compliance, compliance, observability, observability, accuracy, accuracy, accuracy, hallucinations, and hallucinations, and hallucinations, and tone and pitch.

tone and pitch. A total of 26 A total of 26 components that components that components that observe and observe and observe and analyze the behavior analyze the behavior analyze the behavior of the agent as a whole. So of the agent as a whole. So of the agent as a whole. So with this framework, you with this framework, you with this framework, you stop arguing stop arguing stop arguing about whether it's ready about whether it's ready about whether it's ready agent, and translate agent, and translate agent, and translate the discussion the discussion the discussion into facts. You into facts. You into facts. You level the playing field and level the playing field and level the playing field and you can make you can make you can make decisions based on decisions based on decisions based on mathematical rules. mathematical rules. So let's So let's dive into that. dive into that. Each individual Each individual component is scored component is scored component is scored on three levels. on three levels. It gets a zero It gets a zero if the corresponding if the corresponding if the corresponding component is missing, component is missing, component is missing, doesn't exist. It gets a doesn't exist. It gets a doesn't exist. It gets a one if we one if we one if we can measure it.

can measure it. can measure it. We can perceive We can perceive We can perceive it, observe it, observe it, observe it and we can measure it and we can measure it and we can measure it. Not necessarily it. Not necessarily it. Not necessarily automatically. So I automatically. So I automatically. So I can observe can observe can observe it and measure it it and measure it it and measure it manually, that's just as manually, that's just as manually, that's just as good. For example, is good. For example, is good. For example, is the risk identified, what the risk identified, what the risk identified, what is the risk? I can is the risk? I can is the risk? I can read it. I know it. read it. I know it. And the best And the best outcome gets a outcome gets a outcome gets a two if it's two if it's two if it's managed. Managed managed. Managed managed. Managed means it's means it's means it's tracked, tracked, tracked, observed and observed and observed and analyzed over time analyzed over time analyzed over time because everything is because everything is because everything is dynamic. What dynamic. What dynamic. What happened yesterday could be happened yesterday could be happened yesterday could be different from different from different from today. And today. And today. And to accumulate all to accumulate all to accumulate all these different metrics, we these different metrics, we these different metrics, we weight them based weight them based weight them based on importance. And you'll on importance. And you'll on importance. And you'll notice that the notice that the notice that the management cluster is the management cluster is the management cluster is the most important one, most important one, most important one, because I'm analyzing the because I'm analyzing the because I'm analyzing the agent's performance agent's performance agent's performance from a business perspective.

from a business perspective. They're delivering the They're delivering the expected business expected business outcome. So the outcome. So the outcome. So the governance group governance group governance group gets 39%. Next, gets 39%. Next, gets 39%. Next, mandate, and we'll see mandate, and we'll see mandate, and we'll see what that means, what that means, what that means, gets 26%. Platform is gets 26%. Platform is 12, outcome is 9, and 12, outcome is 9, and 12, outcome is 9, and so on. And you'll so on. And you'll so on. And you'll notice that the technical notice that the technical notice that the technical aspects are measured aspects are measured aspects are measured and taken into account, but they and taken into account, but they and taken into account, but they have less weight have less weight have less weight because we're because we're because we're focusing on the focusing on the focusing on the business effectiveness of the business effectiveness of the business effectiveness of the agent. Based on agent. Based on agent. Based on this matrix, we this matrix, we this matrix, we can see that the "green can see that the "green " option, which " option, which " option, which initially looks like initially looks like initially looks like a winner, a winner, a winner, is based on is based on is based on facts and can give a facts and can give a facts and can give a completely different completely different completely different result when we result when we result when we measure all these measure all these measure all these components. Let's take components. Let's take components. Let's take a little look a little look a little look inside each inside each inside each cluster. Governance cluster. Governance is extremely is extremely is extremely important, and it's often important, and it's often important, and it's often forgotten at the very forgotten at the very forgotten at the very beginning. We beginning. We beginning. We focus on the focus on the focus on the technical aspects, and technical aspects, and technical aspects, and only then do we assess the only then do we assess the only then do we assess the business impact. With this business impact. With this business impact. With this framework, we framework, we framework, we start with start with start with that. It also that. It also that. It also helps us a lot in the helps us a lot in the helps us a lot in the planning phase planning phase planning phase when we choose the when we choose the when we choose the functional area functional area functional area to implement AI.

to implement AI. In my projects, I In my projects, I literally see literally see literally see departments departments departments competing to competing to competing to be the first because they be the first because they be the first because they have a directive from have a directive from have a directive from management management management to implement AI. So to implement AI. So to implement AI. So they all want to they all want to they all want to do that, and the framework do that, and the framework do that, and the framework helps them helps them helps them decide. So decide. So decide. So when I define when I define when I define an agent, I start by an agent, I start by an agent, I start by setting the setting the setting the limits. What is the acceptable limits. What is the acceptable limits. What is the acceptable margin of error? You could margin of error? You could margin of error? You could say zero. I say zero. I say zero. I want a perfect want a perfect want a perfect result. I want 90%. result. I want 90%. result. I want 90%. We know that AI is not We know that AI is not We know that AI is not always perfect, but always perfect, but always perfect, but sometimes 50% is good. sometimes 50% is good. For autonomous For autonomous agents, I want a higher agents, I want a higher agents, I want a higher goal, because there is no goal, because there is no goal, because there is no human in the process. human in the process. human in the process. The result has to be The result has to be The result has to be perfect, otherwise perfect, otherwise perfect, otherwise the task the task the task is canceled. But in a is canceled. But in a is canceled. But in a human- human- human- AI collaboration, I know that the AI collaboration, I know that the AI collaboration, I know that the final decision final decision final decision is made by the human. The AI is made by the human. The AI is made by the human. The AI creates the content, and creates the content, and creates the content, and then the human decides.

then the human decides. then the human decides. So 60% margin of So 60% margin of So 60% margin of error can be error can be error can be acceptable. Risks. acceptable. Risks. This is a defined risk. I work in synergy with I work in synergy with other other other departments, so I departments, so I departments, so I know that my agent has to know that my agent has to know that my agent has to work according to work according to work according to this risk plan. this risk plan. I need to I need to check somewhere that these check somewhere that these check somewhere that these risks are defined. And risks are defined. And risks are defined. And if you look at if you look at if you look at the numbers, the agents that we the numbers, the agents that we the numbers, the agents that we measure here measure here measure here show 97% quality, show 97% quality, show 97% quality, because they have a defined because they have a defined risk compliance. Compliance is good. We assessed is good. We assessed is good. We assessed that the agent is compliant with that the agent is compliant with that the agent is compliant with SOC 2 policies and SOC 2 policies and SOC 2 policies and internal rules, internal rules, internal rules, so we got a so we got a so we got a high score. high score. Observability is also good. I can see that also good. I can see that also good. I can see that the agent is logging everything. From the agent is logging everything. From the agent is logging everything. From input, logic input, logic , decisions, to , decisions, to , decisions, to integration calls—everything integration calls—everything integration calls—everything is logged. Good is logged. Good is logged. Good result on that result on that result on that . Drift.

. Drift. Good result. We're Good result. We're still within the still within the still within the boundaries set boundaries set boundaries set at the date of the first at the date of the first at the date of the first launch. That's launch. That's launch. That's measured and measured and measured and tracked. tracked. Accuracy. 3% hallucinations. Some hallucinations. Some hallucinations. Some of that is determined of that is determined of that is determined automatically. Some of it is automatically. Some of it is automatically. Some of it is through through through manual annotation, manual annotation, manual annotation, but overall accuracy but overall accuracy but overall accuracy is measured and is measured and is measured and tracked. And tracked. And tracked. And also tone and mood, also tone and mood, also tone and mood, because these agents are because these agents are because these agents are conversational, I track conversational, I track conversational, I track and extract tone and and extract tone and and extract tone and mood from the conversation. mood from the conversation. So, overall you So, overall you see how many, seven see how many, seven see how many, seven very important metrics are very important metrics are very important metrics are measured. measured. Agent mandate is extremely important. extremely important. extremely important. I know we have a I know we have a I know we have a system prompt system prompt system prompt that defines the boundaries, that defines the boundaries, that defines the boundaries, but that's only part of it. but that's only part of it.

What is the definition of an What is the definition of an agent? What is this agent agent? What is this agent agent? What is this agent supposed to do? Other things supposed to do? Other things supposed to do? Other things from unlimited scope an from unlimited scope an from unlimited scope an agent should not agent should not agent should not perform. So, perform. So, perform. So, hallucination control hallucination control . What is the business impact? . What is the business impact? . What is the business impact? Why are we building Why are we building Why are we building this agent? We are not this agent? We are not this agent? We are not just just just experimenting with experimenting with experimenting with technology. I technology. I technology. I expect this agent expect this agent expect this agent to deliver 70% to deliver 70% contract renewal. That is the ROI. We contract renewal. That is the ROI. We measure that at the end measure that at the end measure that at the end to assess to assess to assess business success. business success. What are the policy boundaries? What are the policy boundaries? What facts What facts What facts should my agent draw should my agent draw should my agent draw conclusions from? conclusions from? conclusions from? Isolated context Isolated context . These are documents, these are . These are documents, these are . These are documents, these are connected systems. connected systems. Security. What Security. What systems is this systems is this systems is this agent allowed to agent allowed to agent allowed to interact with? Very interact with? Very interact with? Very important. I want to sleep important. I want to sleep important. I want to sleep soundly knowing that soundly knowing that soundly knowing that my agent my agent my agent is only accessing is only accessing is only accessing these three endpoints. these three endpoints.

I am not going to give him access I am not going to give him access to an MCP server with 4,000 to an MCP server with 4,000 to an MCP server with 4,000 tools without tools without tools without knowing what he is doing. knowing what he is doing. Isolated context . And secondly, who is . And secondly, who is . And secondly, who is allowed allowed allowed to interact with my to interact with my to interact with my agent? And last agent? And last agent? And last but not least, I but not least, I but not least, I want to stay within want to stay within want to stay within budget. I don't want budget. I don't want budget. I don't want surprises when surprises when surprises when my agent my agent my agent spends 20k on spends 20k on spends 20k on some resources. So some resources. So some resources. So all five of these pillars all five of these pillars all five of these pillars create the create the create the agent's mandate. agent's mandate. agent's mandate. Platform capabilities. We want Platform capabilities. We want Platform capabilities. We want agents to work agents to work agents to work together. We don't want together. We don't want together. We don't want agents to agents to agents to reinvent the wheel reinvent the wheel . If one agent is . If one agent is . If one agent is doing well because they're doing well because they're doing well because they're managing managing managing memory, handoff, memory, handoff, memory, handoff, and and and orchestrating skills orchestrating skills , I want all , I want all , I want all agents to be doing the agents to be doing the agents to be doing the same. So I same. So I same. So I measure that as well.

measure that as well. Handoff is very important. I don't want Handoff is very important. I don't want each agent to be handling each agent to be handling each agent to be handling their own handoff. I their own handoff. I their own handoff. I want to have a want to have a want to have a central hub central hub central hub where people where people where people interact with all the interact with all the interact with all the agents in my agents in my agents in my organization. So I organization. So I organization. So I measure whether measure whether measure whether each each each individual agent is communicating with that individual agent is communicating with that individual agent is communicating with that central service. I central service. I central service. I get into the get into the get into the technical stuff that technical stuff that technical stuff that impacts the impacts the impacts the overall score, but it's overall score, but it's overall score, but it's all all all driven by driven by driven by governance and business governance and business goals. So, yes, goals. So, yes, goals. So, yes, the skills are defined, the skills are defined, the skills are defined, but how but how but how detailed are they? How detailed are they? How detailed are they? How easy are they to find? easy are they to find? easy are they to find? How How How orchestrated are they? orchestrated are they? orchestrated are they? How narrow or How narrow or How narrow or broad is the prompt that broad is the prompt that broad is the prompt that connects multiple connects multiple connects multiple skills? Do skills? Do skills? Do the skills come from a the skills come from a the skills come from a central hub? central hub? central hub? repository, repository, repository, are they built locally are they built locally are they built locally or are they or are they or are they human-coded? What human-coded? What human-coded? What tools are available tools are available ? You'll see ? You'll see that ultimately the best agent is not the one with the is not the one with the is not the one with the most most most tools. It's the one tools. It's the one tools. It's the one who who who uses the uses the uses the tools effectively. So that's tools effectively. So that's tools effectively. So that's also very also very also very important. And that's where I'm at important. And that's where I'm at important. And that's where I'm at .

. But what happens But what happens during execution? during execution? during execution? I need to keep I need to keep I need to keep logs. I need to logs. I need to logs. I need to create tasks create tasks create tasks based on the mandate and based on the mandate and based on the mandate and then execute them. then execute them. Are Are the tasks documented? Are the the tasks documented? Are the the tasks documented? Are the inputs captured? Is there the inputs captured? Is there the inputs captured? Is there the necessary necessary necessary context? How do I context? How do I context? How do I know? Are know? Are success criteria created with the success criteria created with the task? Because task? Because task? Because if I see that an agent if I see that an agent if I see that an agent has renewed a contract, I has renewed a contract, I has renewed a contract, I want to measure whether want to measure whether want to measure whether that renewal was that renewal was that renewal was successful. If I successful. If I successful. If I don't have don't have don't have success criteria associated success criteria associated success criteria associated with that particular with that particular with that particular contract, it's impossible to measure contract, it's impossible to measure contract, it's impossible to measure that. that. So the agent has to So the agent has to explain everything that explain everything that explain everything that they do. That's the they do. That's the they do. That's the subject of the task. And subject of the task. And subject of the task. And at the end, I want at the end, I want at the end, I want to check the result to check the result . There's a single score . There's a single score . There's a single score that determines whether the that determines whether the that determines whether the agent was successful. So, agent was successful. So, agent was successful. So, summing up, summing up, combining all these combining all these metrics, Agent A metrics, Agent A metrics, Agent A gets 83%. If gets 83%. If gets 83%. If you remember, on At you remember, on At you remember, on At the beginning, his the beginning, his the beginning, his retention rate was 80%. There were retention rate was 80%. There were retention rate was 80%. There were 20 escalations. Agent B 20 escalations. Agent B 20 escalations. Agent B only had four only had four only had four escalations. However,

escalations. However, escalations. However, when you look at and when you look at and when you look at and measure governance measure governance , mandate, and bottom , mandate, and bottom , mandate, and bottom line, the final line, the final line, the final score is in favor of Agent score is in favor of Agent score is in favor of Agent A. And if you look A. And if you look A. And if you look at the clusters, in at the clusters, in at the clusters, in every single every single every single cluster, A was better cluster, A was better cluster, A was better than B except for skills. B than B except for skills. B than B except for skills. B had more skills, had more skills, had more skills, but it lacked but it lacked but it lacked explainability, explainability, explainability, tracking success, tracking success, tracking success, and compliance with and compliance with and compliance with governance requirements. governance requirements. So, overall, this So, overall, this system defines a system defines a system defines a real real real winner. Michael, I'll give it back winner. Michael, I'll give it back winner. Michael, I'll give it back to you. to you. to you. Thank you. Thank you, Dan. Thank you. Thank you, Dan. So these seven pillars and the So these seven pillars and the evaluation system are evaluation system are our framework. our framework. our framework. It's a framework that we've It's a framework that we've It's a framework that we've used with used with used with a lot of our a lot of our a lot of our clients, and it clients, and it clients, and it works really well for us works really well for us works really well for us . And clients . And clients . And clients usually tell us, usually tell us, usually tell us, "Hey, they finally have "Hey, they finally have "Hey, they finally have a system in place where a system in place where they can they can measure the measure the measure the performance of the agents performance of the agents they're planning they're planning they're planning and have an intelligent and have an intelligent and have an intelligent discussion with their discussion with their discussion with their business business stakeholders."

stakeholders." So, this is where the So, this is where the intern metaphor intern metaphor intern metaphor probably doesn't work probably doesn't work , right? To your , right? To your , right? To your advantage. When you advantage. When you advantage. When you hire a person, hire a person, hire a person, the interview the interview the interview is over, is over, is over, right? But when you right? But when you right? But when you hire an agent, it hire an agent, it hire an agent, it never never never ends. You ends. You ends. You run run run them on live them on live them on live traffic every week, comparing them to traffic every week, comparing them to the person's baseline: the person's baseline: tooling, tooling, tooling, scoring, improving, scoring, improving, scoring, improving, retesting. retesting. retesting. Again, Again, Again, going back to what going back to what going back to what Dan said earlier Dan said earlier Dan said earlier about drift, right? The green about drift, right? The green about drift, right? The green line you see line you see line you see on the right is the on the right is the on the right is the Vanny dashboard, which has Vanny dashboard, which has Vanny dashboard, which has been consistently at 92 been consistently at 92 % all year. The turquoise % all year. The turquoise % all year. The turquoise line is the actual line is the actual line is the actual score, which drops from 83 score, which drops from 83 score, which drops from 83 to 60 because you're to 60 because you're to 60 because you're measuring it every week. measuring it every week. measuring it every week. So, same So, same So, same agent, same green agent, same green agent, same green dashboard, but different dashboard, but different dashboard, but different metrics over time metrics over time . So one of the things, . So one of the things, . So one of the things, especially when you especially when you especially when you start start start engaging with business engaging with business stakeholders, stakeholders, stakeholders, like your like your CFO, the first CFO, the first question that question that question that comes up is ROI, comes up is ROI, comes up is ROI, right? And then the right? And then the right? And then the fear comes in.

fear comes in. fear comes in. The fear is that the The fear is that the The fear is that the only thing you can do is only thing you can do is only thing you can do is measure in the agent measure in the agent measure in the agent you're working on you're working on you're working on and implementing and implementing and implementing is: "Whose job is it is: "Whose job is it is: "Whose job is it taking away, right?" And taking away, right?" And taking away, right?" And that fear lives on because we have that fear lives on because we have that fear lives on because we have n't measured the alternative for the most part n't measured the alternative for the most part . Here's how . Here's how we try to we try to we try to rephrase it in our rephrase it in our rephrase it in our work with our work with our work with our clients. A good clients. A good clients. A good intern lightens your intern lightens your intern lightens your load, the load, the load, the mundane work mundane work mundane work disappears from your desk disappears from your desk , and you become more , and you become more , and you become more efficient at what's efficient at what's efficient at what's left, right? left, right? left, right? That's the same test for an That's the same test for an That's the same test for an agent. What did they take off agent. What did they take off agent. What did they take off your shoulders? And what did they your shoulders? And what did they your shoulders? And what did they allow your allow your allow your business to do with the business to do with the business to do with the freed up freed up freed up time, right? An agent time, right? An agent time, right? An agent checks the balance at 7:00 checks the balance at 7:00 , at 9:00, so the banker , at 9:00, so the banker , at 9:00, so the banker spends the next spends the next spends the next morning with a client morning with a client morning with a client reviewing their reviewing their reviewing their mortgage. The agent made mortgage. The agent made mortgage. The agent made the banker better at the banker better at the banker better at banking, and that's the banking, and that's the banking, and that's the column that should be on column that should be on column that should be on your scorecard, your scorecard, your scorecard, and that's what we're and that's what we're and that's what we're working on with our working on with our working on with our clients. Now, clients. Now, clients. Now, a confession about the introduction. The a confession about the introduction. The intern in intern in question was never question was never question was never hypothetical, that hypothetical, that hypothetical, that 80/20 ratio 80/20 ratio 80/20 ratio was the benchmark for was the benchmark for was the benchmark for AI adoption in AI adoption in AI adoption in financial services in financial services in financial services in 2026. So everything 2026. So everything 2026. So everything you just heard is you just heard is you just heard is backed up by our backed up by our backed up by our anonymized anonymized anonymized aggregated aggregated aggregated AI agent performance data AI agent performance data . Same metric, . Same metric, . Same metric, deterrence score deterrence score , four different industries, , four different industries, , four different industries, four different four different four different correct answers.

correct answers. correct answers. Financial services—80, Financial services—80, Financial services—80, healthcare—87, healthcare—87, healthcare—87, HR and IT—93, higher education HR and IT—93, higher education —99 in —99 in —99 in deterrence. When you deterrence. When you deterrence. When you look at this data, look at this data, look at this data, higher education is not better higher education is not better higher education is not better than banking. Banking than banking. Banking than banking. Banking simply has simply has simply has more more more redirects in redirects in redirects in its AI its AI agent design. Here's what you're agent design. Here's what you're agent design. Here's what you're weighing: weighing: weighing: each agent's performance against each agent's performance against each agent's performance against one one one universal number universal number . This number is different for . This number is different for . This number is different for different industries. This is different industries. This is Druid's 2026 AI adoption benchmark. We've Druid's 2026 AI adoption benchmark. We've collected 15 months of collected 15 months of collected 15 months of AI agent performance data AI agent performance data AI agent performance data , and it , and it , and it has a lot of has a lot of has a lot of useful data useful data useful data across industries. The report is across industries. The report is free to download. And if you free to download. And if you look at the look at the look at the next slide, right next slide, right next slide, right here, you can here, you can here, you can scan the QR code and scan the QR code and scan the QR code and download the report. And download the report. And the framework we've developed is to the framework we've developed is to define the mandate, define the mandate, define the mandate, set boundaries, set boundaries, set boundaries, evaluate the agent's performance, evaluate the agent's performance, evaluate the agent's performance, and and and compare weekly. with compare weekly. with compare weekly. with human results.

human results. And this is the framework And this is the framework that we usually that we usually that we usually work with our work with our work with our clients. So we clients. So we clients. So we wanted to introduce wanted to introduce wanted to introduce it to all of you, it to all of you, it to all of you, share this share this share this framework, it's framework, it's framework, it's free to free to free to use. And one use. And one use. And one last thing: we're last thing: we're last thing: we're at at at booth LG 11, right booth LG 11, right booth LG 11, right across from Microsoft across from Microsoft across from Microsoft and Open AI. So come on in, and Open AI. So come on in, and Open AI. So come on in, I'd love I'd love I'd love to talk to to talk to to talk to each and every one of you. each and every one of you. each and every one of you. Thank you very much. Thank you very much. Thank you very much. Thank you.