Building the AI Agent Is Easy. Proving It's Safe Isn't - Jonathan Osborne, NatWest

[music] Hi guys. So my name is Jonathan Osborne. I am a lead AI engineer from that west group. I'll tell you more about what that is in a second. And today I'm going to be talking about taking agents from prototype to production customerf facing that is in a regulated banking environment. And I guess maybe some spoilers with the access to coding assistants and tools that we have today. The actual creation of the agent that's kind of the easy part. The much harder part I think is proving that it's safe. So let me tell you a bit about Nat West Group. So Nat West is the second largest retail bank in the UK and currently we have somewhere in the range of 19 million retail customers. 19 million. Hold on to that number because it's really what makes everything else in this talk matter. Kora is our customer's primary point of contact. So it's our digital assistant and it serves somewhere in the realm of a 100,000 conversations a day. And until now the vast vast bulk of those answers were scripted. So if a customer comes to us and they ask a conversation then if we haven't fought to script that then uh you know we're not going to be able to answer. Customers going to get frustrated. They're probably going to have to cue to speak to a human agent.

That's changing. So as of this year we are now working to bring approximately a third of the journeys within Kora to AI generated answers backed by Rag and our knowledge bases and agents agents that are able to take action on the behalf of customers. So for example helping them make a payment or change their address. I guess the scary part here really isn't the model like by and large models are quite good. They provide good answers. Now it's the blast radius. So with 19 million customers, if we get a few of these utterances wrong, that isn't a bug ticket. That isn't kind of a maybe a funny headline, it's a real customer and they're real money. It's slightly very difficult question from the regulator and it's reputational damage for us as a company. So this talk is about what we had to go through to get the approval to go live and I guess what it took us to maybe trust some of our own metrics as well because as I say the interesting part it's not the agent it's proving and making it safe.

Let's maybe make that more concrete by considering an example here. So kind of same question two very different conversations. So customer comes, they ask, "Can I overpay my mortgage without being charged?" Today in the scripted solution, that's going to recognize that keyword mortgage, and we're going to present the customer with a menu. So apply rates, my mortgage, customer is going to have to select none of the above. Maybe we'll ask them to rephrase their question, but more than likely, game over. This customer is going to have to cue, speak to a human agent. They're going to be frustrated. On the right hand side, bit of a different story. So AI has been able to answer that question using our knowledge bases. You can overpay up to 10% of your balance in a year. No early repayment charges. Shall I check your deal? But then the customer does something which is very normal but is can be frustrating. So they change this topic. So actually can I close the account instead? This happens all the time and a traditional decision tree is really really bad at this type of digression. It's worth mentioning that digressions are going to come back. So we'll hold on to that point. I guess the key point though is if we build an agent that can answer any question, it can also answer any question wrong. So that 10% if that doesn't apply to this particular customer's deal, it's a real customer, is their real mortgage, and is there real money at stake?

So why is it so much harder to make an agent back solution, you know, more, you know, how do we protect against it as opposed to that scripted bot? Well, it's because it reasons, it decides, it generates. And there are at least kind of four failure mode here, which we wouldn't have in that scripted solution. One, it can be confidently wrong. Hallucination against a fictitious source and sound absolutely sure of itself while it does it. Two, it can be talked into things, prompt injection. The input itself can become adversarial. Three, right, but unhelpful, so it can provide accurate on policy answers that just go round and around and around and never actually answer the customer's conversation. incredibly frustrating. And four, it can be nondeterministic. Same inputs, completely different output. That means that your traditional software engineering assertions just don't really work very well here. And as a consequence of that, regression tests that have been built up over years by the existing core team, they really no longer apply. A scripted systems failure surface is completely bounded. We can write out every situation that might occur. we can test it and we can be really confident it's going to do what we want it to with an agent that's really not possible and we can't really traditionally test our way to safety. So how do we deal with this? Well, the first part for us was shrinking the attack surface before we actually then thought about hardening it introducing what we call the routing funnel. So every customer conversation comes in at the top here. First we apply a contextual filter and that allows us to

remove kind of common events like button clicks. That's about 20%. Next, we have some deterministic pre-classification rules. So, with these rules, we're kind of using the customer's utterance and some metadata to route them to a destination with no ML or AI involved at all. At this point, it's only utterances that make it past these first two gates that we then actually start the models on. First, there's a relevancy and rephrasing step that's kind of gated. It helps with those really detailed multicontext questions. And then our intent classifier. This is actually a neural network over embeddings. This isn't generative AI which is able to use a customer's utterance to predict the right back end. Finally, there's some policy rules to ensure that this customer should actually be getting into that particular backend. Why do we build it this way? Well, four reasons a lot of them about being a bank. Firstly, access control. So, we need to enforce that to ensure the customer should be getting to this act this this back end. Auditability. So, every decision here needs to carry a code. It needs to be fully auditable and reproducible. The regulator ensures this resilience. In the case that a model or the gender isn't available, we still need to provide the customer an answer and I guess governance. So all of this is version config. We can make changes to it without a full redeployment.

That's really important because it means that we can react fast to changing market conditions. We didn't just take a language model, slap some context into it, and point it at 19 million customers. We first thought really hard about what actually needed a model to solve. This brings us on to the really important question of is this actually working? And the key metric that they looked at in kind of traditional Kora was accuracy when doing this. So if I told you this intent writer is 94% accurate, that probably sounds really good. But does that actually answer the important question? So is it grounded in a real source? Did the customer's question actually get resolved? Did it know when it was out of its depth? When to stop bring in a human? You can't embody that with one single score. And that's why we've moved away from quality as a score, started treating it more as a vector. Uh, every score, every kind of answer we score across three different categories. So, safety, could it harm the customer? Task, did it actually do its job? Experience, how did it feel to deal with? All of these can be monitored and alarmed independently. And they allow us to differentiate between is it hallucinating versus is it right but just insufferable to deal with. There are different answers and they require really different solutions.

It's what the key point here is a routter that's 94% accurate but 100% confident the 6% of the time it's wrong is way more dangerous than one that's 85% confident and knows it. The first thing we did here before we worried about automating this is what we always do and we started off with some human validation. Uh the numbers that you see here are some June validation that we did on our payment agent. 111 conversations over 37 different payment journeys. Four brands, three independent human scorers per conversation. Headline here, 83% pass rate across the board. But as ever, the headline's pretty useless. Let's dig into some of that data. So 97% at on the happy path scenarios. That's great. 76% edge cases, not quite so good. Topic changes like that mortgage example that we discussed earlier, only 43%. So that single 83% score was hiding that 54% spread. The second point on the right is probably more nuanced and less talked about and that's do human validators actually agree with each other [snorts] on something objective like bias and fairness you do actually see a 95% agreement rate which is great but on something more subjective intent understanding so you know are does the bot actually understand what they what the user wants only a 64% agreement rate. The common thing when we start automating this bringing LLM as a judge is that we want to make our judge match our human validators. But what are they meant to be matching here? There's only a 60 form agreement rate between humans.

So that can't simply be the target here essentially. One one caveat by the way like the two biggest causes of failure here actually turned out to be engineering issues. Not problem with the model at all, but nevertheless that proves useful value of the human validation because it caught them. So let's turn to actually automating this then. So LLM as a judge. So core plus that's not live yet. So I'm going to turn to another internal project. So this is an internal complaints agent that we have. Essentially the goal here is that customers will occasionally complain. This will take the complaint. It has access to a load of tools to then be able to write a summary for complaint handlers to essentially explain why has this particular user complained and how we can help them with it. It went live in February and we've got around 14,000 of these cases which have been scored by both a human validator and an LLM as a judge. If we dive into the data, so this is the faithfulness score here on the right. So you can see upper bar in the red there, that's the LM as a judge mean.97 right up at the top very little spread.

Human mean 083 much much wider spread. Key point here is that if you look at that average error versus the human it's 2. you might conclude that the judge is doing okay but the correlation is almost zero. Uh in in this case like a judge which doesn't discriminate between the actually good cases and the bad cases is basically worthless. You don't want your judge to be generous and that is what LLMs want to do by default. Maybe one other point to make here is that so that amber dotted line that represents the threshold for this particular metric for this judge model. You can see we're comfortably above that. However, we made a change to that model. Didn't change anything else. Same inputs, same outputs. Just change the actual model behind it. And that dropped all the way from point 8 above to 2 above. That means that the judge model itself actually becomes part of your configuration. And it can lead to breaking changes. A key point we took here, by the way, is that we moved our judge to a different model family from that of the uh, you know, the actual agent. So, judged and judged, different blind spots, and that's actually been really effective in pinning the two scores closer together.

Coming back to the core example. So how does this scoring actually run? So live conversations between the agents and its guardrails. All of those are captured ving open source standards like hotel and open inference. All of that detail is then sampled. So by default we take 5% random sample but we also want to look at everything interesting. By interesting I mean conversations are handoff fail those are particularly long ones where the user is angry. We then score these with our LLM as a judge. So that's that raft and dimension of metrics that I talked about earlier. And then absolutely all of this ends up in comedic. So traces, scores, experiments. It's governed by a couple of bank constraints here. So we work in a completely airgapped environment. So no internet access to anything. So judge model is reached over a private path. We have an internal geni gateway. Comet a SAS product. We're consuming that as a signed partner app via AWS. Second around personally identifiable information. We have a zero tolerance policy for any PII leaving our network. So that means a lot of this evaluation happens without that present different cadences here. So tracers are being captured as they happen. We're doing offline scoring every four hours. And then regress this is also built into our CI/CD. So if we make changes to prompts or to context that can have like an outsized impact on what the agent is actually responding. So we need to constantly be comparing versus a known good baseline. We do that in CI/CD.

There's a hard regression gate. So changes can't be merged in without us understanding exactly what's going to occur. I guess key point is that continuous evaluation is part of your production infrastructure. It's not a notebook that you run a couple of times when your model goes live. So there's different models here and they've got different failure modes. So on the right hand side we've got the agents, we've got the generative AI. So generative AI regresses. So as we've just talked about, somebody can make changes to a prompt and that can have outsized impact on the behavior. That's why we have the CI/CD in place. You can see here a real opic trace which is looking from our payment agent. Every node in the LAN graph uh becomes a different span and when the agent is slow or is breaking in a particular place, this allows us or an agent to jump in and investigate exactly why on the classical. So the intent classifier is traditional ML and thus we've got to monitor it in a more classical way. It drifts over time, right? So human conversation will naturally evolve from the point at which the model was trained. So we need to look at more traditional things like input feature drift, F1 score based off human scores, sorry, human labels. And then you know we have rag thresholds defined as to exactly when we need to trigger retraining for that model. On the right you can see an npm data drift chart. I I really wouldn't underestimate like input feature drift here. It's a really good canary as to when your model is starting to decline before your F1 score does implying the need for retraining. So two different problems, two different instruments, but what we don't want is two different observability and evaluation frameworks. Comet's really

useful there because it brings in both OPIC for the Genai and MPM model performance monitor for the ML. So one platform how do we actually deploy this then without kind of breaking the bank and creating a load of headlines. We start off with a shadow deployment. So what I mean by that is we put this live uh we use it to start classifying cutter utterancerances but this existing scripted solution is what will actually respond to the customer and that means that we can get a sense of the real data you know the the input uh the input space for different customer queries with almost zero risk. I think this shadow mode is like one of the more transferable bits of this project for us because often we have this deadlock in a regulated industry where obviously we can't go live and get production data without the approval to do so. But equally to actually enable us to go live in many cases our model risk colleagues increasingly want to see how our models are going to react to that actual production data. This allows us to kind of break that apart so we can break apart observe safely from actually answer customer queries.

uh and it takes one really big decision point, a big bang release into a series of smaller ones which we can make confidently. We then go brand by brand. So we have like percentage based rollout within the brand and we have kind of KPIs which allow us to determine go no-go at various points along there. Some of those decisions are automated and some of those require manual approval. Um I think and yeah some of those require manual approval. What kind of makes that possible is that we make a lot of use of feature flagging here. So there's like a global flag that we can trigger to send everything back to the scripted solution, but also every single backend can be isolated either globally or on a particular dimension such as brand and all of that can be done in seconds. Before I wrap up, let me maybe just talk very briefly and pull this back to that complaints project. So just to remind you that was the colleague facing agent which writes investigation summaries for summaries for complaint handlers. In that project, human feedback is a gate. It's not a thumbs up. It's not a tickbox activity. So if a complaint handler wants to rerun their investigation, they can't do so until they provided human feedback. We want to understand why they felt the need to do that rerun. But also it's allowed us to build up this quality data set of 14 14,000 customer of uh of colleague labels I guess in this case and that's really valuable in a lot of projects. We struggle to get a tenth of that number and I think the projects have also traded nicely here. So you know they found that judge problem in production we have judge calibration and regression checks because of that but

equally the architecture that we built which relies heavily on commodopic they don't use that yet and they're going to be adopting that for their production monitoring I guess it's a glimpse inside an enterprise right so the the agent was the cheap part here that's code that we can produce to solve the business problem the valuation spine is the bit that we can then reuse again and again so to wrap up like if you are going to be shipping customerf facing agents in a regulated environment, then these are probably the points I would give you. I'm not going to go through everything. I'll maybe just pick a couple. So, three and four here, they're the they're the points with the actual production judge data, right? So, that judge pinned near the top, it's a broken instrument. Your judges are going to want to be overly positive. You need to fight against that. Compare the spans, not just the averages. Ensure the judge is actually matching the human reviewer. Fourth, the judge model itself does become part of that threshold config, right? So if you are going to be making changes to the judge model then uh you know you can introduce a breaking change without knowing it.

Comet is obviously absolutely key here. So we have that for both halves both the generative AI and the traditional ML which we do still use a little bit. So and I guess the the final point here is next month I'll find out how much of this is actually true when we go live. So thanks a lot for listening. It was really a pleasure to get to speak to all of you. So I'll be at the comment booth after this. So if you do have any questions, you know, disagreements, etc., really happy to hear them. But thanks a lot for your time.