Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS

Hello, congratulations. Hello, congratulations. Welcome to Welcome to Beyond Done. We Beyond Done. We Beyond Done. We are about to start are about to start are about to start our seminar. We have our seminar. We have our seminar. We have a lot of material, and a lot of material, and a lot of material, and we want to we want to we want to make sure make sure make sure we have time to answer we have time to answer we have time to answer questions and get everyone questions and get everyone questions and get everyone comfortable. comfortable. So, we begin. Regarding the name “Beyond Done”, Regarding the name “Beyond Done”, to get straight to get straight to get straight to the point: you probably to the point: you probably to the point: you probably saw the tweet about that saw the tweet about that saw the tweet about that incident when the agent incident when the agent incident when the agent started deleting started deleting started deleting working databases. In working databases. In working databases. In 9 seconds, everything was 9 seconds, everything was 9 seconds, everything was destroyed. We want to destroyed. We want to destroyed. We want to help you help you help you prevent this prevent this prevent this today by maintaining today by maintaining today by maintaining trust in your trust in your trust in your agents so you don’t agents so you don’t agents so you don’t have to have to have to check every check every check every line or even every line or even every line or even every PR. This is the path PR. This is the path PR. This is the path we think we think we think technology is taking, and how do we technology is taking, and how do we technology is taking, and how do we get there? How do we get there? How do we get there? How do we make this a make this a make this a reality, where reality, where reality, where instead of such disruptions, instead of such disruptions, instead of such disruptions, we have peace of mind, everything we have peace of mind, everything we have peace of mind, everything works, and you can works, and you can works, and you can trust trust trust your agents to a certain extent, your agents to a certain extent, your agents to a certain extent, gradually building gradually building gradually building that trust. that trust. I would also say that it's I would also say that it's not just about not just about not just about preventing mistakes preventing mistakes . Today we . Today we . Today we will look at this as a will look at this as a will look at this as a foundation for foundation for foundation for achieving achieving achieving true true true scalability when scalability when scalability when working with agents.

working with agents. working with agents. So, I'm Nick Nisi. I am an So, I'm Nick Nisi. I am an So, I'm Nick Nisi. I am an engineering manager engineering manager engineering manager at WorkOS. I live in Omas, at WorkOS. I live in Omas, at WorkOS. I live in Omas, Nebraska and Nebraska and Nebraska and have been there for have been there for have been there for about 2 years. I about 2 years. I about 2 years. I work a lot with work a lot with work a lot with agents. I haven't written agents. I haven't written agents. I haven't written code since July 30, 2025. code since July 30, 2025. code since July 30, 2025. I remember because it was the I remember because it was the I remember because it was the day I day I day I came back from Laracon, came back from Laracon, came back from Laracon, saw PHP and said to saw PHP and said to saw PHP and said to myself, "Never again myself, "Never again ." And here I am. ." And here I am. ." And here I am. My best friend My best friend My best friend Zach is with me. Zach is with me. Zach is with me. Hi, I'm Zack Prosser. Hi, I'm Zack Prosser. Hi, I'm Zack Prosser. I am an engineer in the I am an engineer in the I am an engineer in the Applied AI team at WorkOS. I'm Applied AI team at WorkOS. I'm Applied AI team at WorkOS. I'm so happy to be here with so happy to be here with so happy to be here with all of you today in all of you today in all of you today in New York. New York. We've already We've already shown this on screen before. shown this on screen before. shown this on screen before. This repository is your This repository is your This repository is your working tool. working tool. working tool. All slides All slides All slides are in this are in this are in this repository. You repository. You repository. You can find them here. can find them here. can find them here. You can also You can also You can also view them at view them at view them at workos.cloud/beyond-done workos.cloud/beyond-done . I know it sounds . I know it sounds . I know it sounds complicated, but complicated, but complicated, but the following QR code the following QR code the following QR code seems to take seems to take seems to take you there. Sorry, you there. Sorry, you there. Sorry, talks.workos.cloud/beyond-done. I talks.workos.cloud/beyond-done. I talks.workos.cloud/beyond-done. I wanted to emphasize wanted to emphasize wanted to emphasize this because I'm this because I'm this because I'm going to quickly switch to going to quickly switch to going to quickly switch to this page now. All this page now. All this page now. All slides are available there.

slides are available there. slides are available there. There is also a page There is also a page There is also a page for workshops with for workshops with for workshops with exercises that we exercises that we exercises that we will be doing. will be doing. will be doing. So you can So you can So you can walk through each walk through each walk through each part. You can part. You can part. You can work on this at a work on this at a work on this at a time convenient for you. time convenient for you. time convenient for you. You can You can You can check this out check this out check this out later. This is a step-by-step later. This is a step-by-step later. This is a step-by-step guide on how to guide on how to guide on how to do it. There is also a do it. There is also a do it. There is also a glossary covering glossary covering glossary covering all terms, big and all terms, big and all terms, big and small; some of them are small; some of them are small; some of them are familiar to you, familiar to you, familiar to you, hopefully hopefully hopefully most of them are most of them are most of them are familiar to you. But if there are familiar to you. But if there are familiar to you. But if there are any that aren't, you any that aren't, you any that aren't, you can look in the can look in the can look in the glossary to glossary to glossary to review them. There review them. There review them. There is also an "ask is also an "ask is also an "ask anything" button where you can anything" button where you can anything" button where you can chat with chat with chat with the model trained in the model trained in the model trained in this presentation and this presentation and this presentation and hopefully hopefully hopefully get get get answers there. But you answers there. But you answers there. But you can also just can also just can also just chat with a chat with a chat with a person, that is, with us. person, that is, with us. person, that is, with us. That's right. Yes, we'll That's right. Yes, we'll That's right. Yes, we'll leave that on the screen leave that on the screen . We've noticed that . We've noticed that . We've noticed that many people want to many people want to many people want to ask questions ask questions ask questions but for some reason but for some reason but for some reason can't, so be can't, so be can't, so be sure sure sure to check this out and to check this out and to check this out and take advantage of it. take advantage of it. take advantage of it. Steeply. Let's Steeply. Let's Steeply. Let's wait a second. I wait a second. I wait a second. I see some cameras see some cameras see some cameras pointed at QR codes. pointed at QR codes. pointed at QR codes. The security service would be The security service would be The security service would be very angry with us. very angry with us. OK. So when we OK. So when we think about think about think about the completion of work, the completion of work, the completion of work, about agents about agents about agents completing a task, there are completing a task, there are completing a task, there are many factors that many factors that many factors that influence our influence our influence our confidence in that. And confidence in that. And confidence in that. And the very first question: is the very first question: is the very first question: is this code correct?

this code correct? this code correct? Is this the code Is this the code Is this the code I would write? Did I would write? Did I would write? Did he create good PR he create good PR ? And then I review ? And then I review ? And then I review it like a regular it like a regular it like a regular code reviewer. But code reviewer. But code reviewer. But it takes a lot of time it takes a lot of time . And these agents write . And these agents write . And these agents write code much faster code much faster code much faster than us, so we don't than us, so we don't than us, so we don't want to want to want to rely too much on that because it rely too much on that because it rely too much on that because it creates a creates a signal-to-noise problem and we end up with signal-to-noise problem and we end up with too much too much too much code to review. code to review. We no longer write We no longer write code ourselves. And we are code ourselves. And we are code ourselves. And we are less familiar with this less familiar with this less familiar with this code, so we become code, so we become code, so we become more and more more and more more and more removed from removed from removed from it. So, we it. So, we it. So, we need to find other need to find other need to find other ways to do this. And there are ways to do this. And there are ways to do this. And there are many ways to many ways to many ways to do this. You can do this. You can do this. You can use use use browser browser browser tools tools tools like Playwright to like Playwright to like Playwright to say, "Hey, I say, "Hey, I say, "Hey, I have this website, and I have this website, and I have this website, and I submitted a form." submitted a form." submitted a form." The sending worked The sending worked . I tested it and . I tested it and . I tested it and I got a I got a success message. This is a great success message. This is a great writing test. writing test. writing test. There's nothing specifically There's nothing specifically There's nothing specifically agent-specific about this, but agent-specific about this, but agent-specific about this, but it's something agents it's something agents it's something agents can can can use to use to use to check their check their check their work. And they can work. And they can work. And they can report it report it report it themselves. And you themselves. And you themselves. And you can can can trust that. Often, but trust that. Often, but trust that. Often, but not always. And so we not always. And so we not always. And so we want to figure out: how do we make want to figure out: how do we make want to figure out: how do we make sure we sure we sure we can really can really can really trust these reports?

trust these reports? Certainly. So, everything Certainly. So, everything we have prepared for we have prepared for we have prepared for you today in this you today in this you today in this workshop, as workshop, as workshop, as Nick mentioned, Nick mentioned, Nick mentioned, is repository-based and is repository-based and is repository-based and interactive. So feel interactive. So feel interactive. So feel free to free to free to take take take the opportunity to give the opportunity to give the opportunity to give it a go, it a go, it a go, experiment, experiment, experiment, and take away something and take away something and take away something that you might that you might that you might be able to use be able to use be able to use at work or at work or at work or at least try at least try at least try in a different way. in a different way. in a different way. Yes. So if you Yes. So if you Yes. So if you just go to the just go to the just go to the repository like I do and repository like I do and repository like I do and type ` pnpm snack `—if type ` pnpm snack `—if type ` pnpm snack `—if I can I can I can type it, of course—that's why type it, of course—that's why type it, of course—that's why I have I have I have abbreviations for abbreviations for abbreviations for everything. P—this is my ` pnpm`. everything. P—this is my ` pnpm`. everything. P—this is my ` pnpm`. Uh, it'll open the Uh, it'll open the Uh, it'll open the browser and we can browser and we can browser and we can look at it. We look at it. We look at it. We see this see this Midnight Snack Club app that we Midnight Snack Club app that we Midnight Snack Club app that we created. You can created. You can created. You can play with it play with it play with it right now, and I can right now, and I can right now, and I can test it myself. test it myself. test it myself. Let me Let me Let me just enter my just enter my just enter my email address. I'll email address. I'll email address. I'll type in, uh, [email protected]. type in, uh, [email protected]. type in, uh, [email protected]. Thank you, friend. Thank you, friend. Thank you, friend. Yes. And look at Yes. And look at Yes. And look at this. Success. Email this. Success. Email this. Success. Email address saved. address saved. address saved. Email address Email address Email address here. I see her. If here. I see her. If here. I see her. If I had a test to I had a test to I had a test to confirm that confirm that confirm that the address appeared the address appeared the address appeared on the page, everything would on the page, everything would on the page, everything would work fine. So this is what work fine. So this is what these examples look like. You these examples look like. You can see it, uh, can see it, uh, can see it, uh, while you're working. But while you're working. But while you're working. But if you if you if you reload reload reload the page, like this, the page, like this, the page, like this, nothing will remain.

nothing will remain. nothing will remain. So, although...oh, what did I So, although...oh, what did I So, although...oh, what did I do? A. do? A. do? A. Switching back. Uh Switching back. Uh , if you , if you , if you reload this, uh, you reload this, uh, you reload this, uh, you can see that there's no one there can see that there's no one there can see that there's no one there . I can't seem to . I can't seem to . I can't seem to control Chrome. Uh control Chrome. Uh , and that's because it , and that's because it , and that's because it was updated, but the data was was updated, but the data was was updated, but the data was n't saved anywhere. n't saved anywhere. n't saved anywhere. So we didn't have So we didn't have So we didn't have any verification. We're any verification. We're any verification. We're just checking just checking just checking the interface, looking, the interface, looking, the interface, looking, "Oh, did that actually "Oh, did that actually "Oh, did that actually do anything?" And yes, it did do anything?" And yes, it did do anything?" And yes, it did affect the affect the affect the interface, but we didn't interface, but we didn't interface, but we didn't check the other check the other check the other part, so the data part, so the data part, so the data wasn't saved and we wasn't saved and we wasn't saved and we lost that lost that lost that email address. email address. email address. So, sorry, Zach, you're So, sorry, Zach, you're So, sorry, Zach, you're not registered. not registered. not registered. It suits me. Uh It suits me. Uh , yes, I think this is an , yes, I think this is an , yes, I think this is an extremely important extremely important extremely important point. So, uh, point. So, uh, point. So, uh, raise your hands: who in raise your hands: who in raise your hands: who in this room is creating this room is creating this room is creating something with the help of something with the help of something with the help of agents, let's start with the agents, let's start with the agents, let's start with the first question. Okay first question. Okay , great, more than , great, more than , great, more than half the room. Uh, half the room. Uh, half the room. Uh, how many people how many people how many people have have have multiple multiple multiple terminal tabs open, or multiple terminal tabs open, or multiple terminal tabs open, or multiple chat sessions, or multiple chat sessions, or multiple chat sessions, or multiple agents—it seems like everyone agents—it seems like everyone agents—it seems like everyone who raised their hands who raised their hands who raised their hands for the first time, right? And that's for the first time, right? And that's for the first time, right? And that's exactly what we all exactly what we all exactly what we all feel, feel, feel, at least I do, and what at least I do, and what at least I do, and what Nick feels. If this is the Nick feels. If this is the Nick feels. If this is the eighteenth, eighteenth, eighteenth, you know, workstream that you know, workstream that you know, workstream that you're you're you're working on today, and working on today, and working on today, and your attention is your attention is your attention is scattered, so to speak, scattered, so to speak, scattered, so to speak, then these then these then these subtle errors occur because subtle errors occur because subtle errors occur because the agent says, "Yes, I the agent says, "Yes, I the agent says, "Yes, I did everything you did everything you did everything you asked; the code looks asked; the code looks asked; the code looks good, it compiles, it good, it compiles, it good, it compiles, it runs, the tests runs, the tests runs, the tests pass, but it's pass, but it's pass, but it's still still still wrong." And so what wrong." And so what wrong." And so what we're we're we're going to ask going to ask going to ask and explore in this and explore in this and explore in this workshop together is: How do workshop together is: How do workshop together is: How do we actually we actually we actually fix this fix this fix this problem so that the agents that problem so that the agents that problem so that the agents that we're all we're all we're all trying to trying to trying to organize now, like organize now, like organize now, like herds of crazy herds of crazy herds of crazy token machines, can token machines, can token machines, can figure out for themselves what they figure out for themselves what they figure out for themselves what they need to do and need to do and need to do and when they've when they've when they've made a mistake made a mistake ?

? ? Of course. Of course. Of course. And you'll notice that we're And you'll notice that we're And you'll notice that we're not actually not actually not actually looking at the code for looking at the code for looking at the code for this. We won't go this. We won't go this. We won't go too deep too deep too deep into the code because, like I into the code because, like I into the code because, like I said, I've seen PHP and said, I've seen PHP and said, I've seen PHP and I'm scared, so we're I'm scared, so we're I'm scared, so we're not looking at the not looking at the not looking at the code anymore. But all of this code anymore. But all of this code anymore. But all of this brings us to a brings us to a brings us to a great quote from a great quote from a great quote from a famous "potato" in X, famous "potato" in X, famous "potato" in X, her name is Lauren Ten. her name is Lauren Ten. her name is Lauren Ten. She has this quote: She has this quote: She has this quote: If an agent can't If an agent can't If an agent can't verify their own verify their own verify their own work, then nothing else work, then nothing else work, then nothing else matters. matters. matters. So, in other words, and So, in other words, and So, in other words, and I think this is I think this is I think this is supported by my supported by my supported by my experience and the experience of experience and the experience of experience and the experience of others, we agree others, we agree others, we agree that that that verification now, verification now, verification now, especially as especially as especially as the capabilities of models the capabilities of models the capabilities of models are growing rapidly, is are growing rapidly, is are growing rapidly, is one of the key one of the key one of the key fundamental fundamental fundamental things to things to things to focus on, right? focus on, right? focus on, right? It's like reliability, It's like reliability, It's like reliability, verification, and verification, and verification, and evaluation. Especially evaluation. Especially evaluation. Especially as we start to as we start to as we start to see more and more see more and more see more and more tools that tools that tools that have the concept of RSI, have the concept of RSI, have the concept of RSI, or, you know, or, you know, or, you know, recursive recursive recursive self-improvement, self-improvement, self-improvement, it seems like we're it seems like we're it seems like we're moving into a world where moving into a world where moving into a world where we can all eventually we can all eventually we can all eventually maintain multiple maintain multiple self-evolving codebases, self-evolving codebases, right? And in such a right? And in such a right? And in such a world, how do we world, how do we world, how do we lay the foundation for lay the foundation for lay the foundation for evaluation so that the agent evaluation so that the agent evaluation so that the agent truly understands what a truly understands what a truly understands what a quality quality quality result is and can result is and can improve their improve their performance on their own, and how are we performance on their own, and how are we performance on their own, and how are we going to give the going to give the going to give the agent the agent the agent the feedback they need feedback they need feedback they need to understand where they went to understand where they went to understand where they went wrong?

wrong? wrong? Yes, until we Yes, until we Yes, until we solve this, we won't solve this, we won't solve this, we won't be able to trust these be able to trust these be able to trust these agents enough to agents enough to agents enough to fully fully fully accelerate our accelerate our accelerate our work. So we work. So we work. So we need to address this at need to address this at need to address this at different scales, and it's not different scales, and it's not different scales, and it's not just going from just going from just going from zero to one, but a zero to one, but a zero to one, but a whole gradient, and whole gradient, and whole gradient, and getting closer to getting closer to getting closer to one where we one where we one where we can fully can fully can fully trust these agents trust these agents -- that's our goal. -- that's our goal. And I'll also just And I'll also just share that, you know, share that, you know, share that, you know, again, from my again, from my again, from my own own own observations, this is what observations, this is what observations, this is what we all we all we all feel about WorkOS. feel about WorkOS. feel about WorkOS. We have an We have an We have an applied AI team that applied AI team that applied AI team that builds a bunch of different builds a bunch of different builds a bunch of different things for other things for other things for other teams and for teams and for teams and for ourselves, and it's a completely ourselves, and it's a completely ourselves, and it's a completely different experience when you're different experience when you're different experience when you're working with a working with a working with a codebase that has a codebase that has a codebase that has a baseline that you can baseline that you can baseline that you can run and run and run and trust. trust. trust. You know, we'll get You know, we'll get You know, we'll get to some tips and to some tips and to some tips and tricks, but tricks, but tricks, but being able to actually being able to actually being able to actually see if see if see if he did something right he did something right he did something right or not, just like we or not, just like we or not, just like we would do as would do as would do as human developers, human developers, human developers, right? So right? So right? So the metaphor of us the metaphor of us the metaphor of us all becoming all becoming all becoming agent managers agent managers agent managers is quite common. is quite common. is quite common. This is how we ensure This is how we ensure This is how we ensure that each one of them is that each one of them is that each one of them is actually doing the actually doing the actually doing the right job and right job and right job and we are essentially not we are essentially not we are essentially not wasting money and wasting money and wasting money and time. time. time. Yes, that's the equivalent of Yes, that's the equivalent of Yes, that's the equivalent of saying: don't saying: don't saying: don't make mistakes, make mistakes, make mistakes, right? Yes, if you right? Yes, if you right? Yes, if you want to have some kind of want to have some kind of want to have some kind of verification, you can't verification, you can't verification, you can't just just just guess what it should guess what it should guess what it should be. You need to be. You need to be. You need to give him clear, give him clear, give him clear, defined goals to defined goals to defined goals to achieve and achieve and achieve and provide a way to provide a way to provide a way to monitor and monitor and monitor and check those goals.

check those goals. check those goals. If not, the agent If not, the agent If not, the agent will do something, spend will do something, spend will do something, spend tokens, and it may tokens, and it may tokens, and it may get to the get to the get to the result you result you result you expect, but there are expect, but there are expect, but there are no guarantees. no guarantees. no guarantees. For every detail For every detail For every detail you haven't filled in you haven't filled in you haven't filled in regarding the outcome and regarding the outcome and regarding the outcome and what you what you what you really want to really want to really want to get at, he'll just get at, he'll just get at, he'll just draw a conclusion or draw a conclusion or draw a conclusion or make up something of his own. And make up something of his own. And make up something of his own. And this assumption that this assumption that this assumption that he makes may not he makes may not he makes may not match your match your match your assumptions or what assumptions or what assumptions or what you you you expected because you are a expected because you are a expected because you are a rational logical rational logical rational logical person and you just have person and you just have person and you just have n't reconciled it yet. So n't reconciled it yet. So n't reconciled it yet. So you need to have these you need to have these you need to have these systems in place to systems in place to systems in place to allow it to allow it to allow it to observe, observe, observe, understand what you understand what you understand what you want, check it want, check it , correct it, and , correct it, and , correct it, and check it again. check it again. Yeah, and I think, again Yeah, and I think, again , in our own , in our own , in our own observations, we observations, we observations, we all find that the all find that the all find that the outdated way of outdated way of outdated way of doing it is to have a doing it is to have a doing it is to have a huge huge huge PR and one PR and one PR and one agent session that runs for 3 agent session that runs for 3 agent session that runs for 3 or 4 days and say, or 4 days and say, or 4 days and say, fix this, fix this, fix this, fix this, fix this, fix this, make it better. And what make it better. And what make it better. And what works almost works almost works almost perfectly for humans perfectly for humans is a very clear separation of is a very clear separation of is a very clear separation of tasks, giving tasks, giving tasks, giving each agent in each agent in each agent in the request exactly what the request exactly what the request exactly what they need to do they need to do they need to do to know that the to know that the to know that the task is actually task is actually task is actually done, and done, and done, and to check the to check the to check the completion status. And they completion status. And they completion status. And they will just fly.

will just fly. will just fly. I mean, we have I mean, we have I mean, we have people on our team who people on our team who people on our team who are doing 900 pull are doing 900 pull requests a month requests a month requests a month without incidents and without incidents and without incidents and improving improving improving KPIs for the KPIs for the KPIs for the software. So, software. So, software. So, purely from my own purely from my own purely from my own experience, this is something experience, this is something experience, this is something we should all we should all we should all think about: I think about: I think about: I no longer approach this no longer approach this no longer approach this codebase in the codebase in the codebase in the traditional way traditional way . I'm almost like a gardener, . I'm almost like a gardener, . I'm almost like a gardener, tending to tending to tending to multiple multiple multiple codebases and letting them codebases and letting them codebases and letting them grow on their grow on their grow on their own. Yes. own. Yes. And that's what we And that's what we create, and we also create, and we also create, and we also use our use our use our own tools. own tools. own tools. This is how Zach and I This is how Zach and I This is how Zach and I stay stay stay productive in WorkOS productive in WorkOS productive in WorkOS and get our and get our and get our work done. And each of us work done. And each of us work done. And each of us has built our own has built our own has built our own examples of this and examples of this and examples of this and our own approaches to our own approaches to our own approaches to work. And we really work. And we really work. And we really think that's a think that's a think that's a good thing right now. It's too early good thing right now. It's too early good thing right now. It's too early to classify anything, so to classify anything, so to classify anything, so experimentation is the experimentation is the experimentation is the key: find what key: find what key: find what works and build on works and build on works and build on it. And one thing that we it. And one thing that we it. And one thing that we both use both use is verification, or is verification, or is verification, or building everything building everything building everything around this around this around this verification cycle. Because, as verification cycle. Because, as verification cycle. Because, as this quote says, this quote says, this quote says, status is determined status is determined status is determined by records, not by records, not by records, not models. So, you know, models. So, you know, models. So, you know, show me the show me the show me the results. Show results. Show results. Show me real evidence me real evidence me real evidence that everything that everything that everything worked as I worked as I worked as I expected. Don't just expected. Don't just expected. Don't just tell me, "Yeah, tell me, "Yeah, tell me, "Yeah, it's going, it's going, it's going, it looks good." Because there is it looks good." Because there is it looks good." Because there is nothing to nothing to nothing to confirm this.

confirm this. confirm this. I need I need I need screenshots, videos, screenshots, videos, screenshots, videos, test results, test results, test results, smoke tests, any of the smoke tests, any of the smoke tests, any of the above to above to above to prove to me without prove to me without prove to me without looking at the code that looking at the code that looking at the code that it actually does what it actually does what it actually does what I want. Because I want. Because I want. Because in the end, I in the end, I in the end, I care less about the code. I care less about the code. I care less about the code. I care about the result, what care about the result, what care about the result, what it does and what it does and what it does and what value it brings. value it brings. value it brings. Honestly, there Honestly, there Honestly, there may be bad code may be bad code that still produces a that still produces a that still produces a pretty good pretty good pretty good result. Or there may result. Or there may result. Or there may be very good code be very good code that doesn't give that doesn't give that doesn't give me the desired me the desired me the desired result at all. And I result at all. And I result at all. And I need to find need to find need to find the right balance the right balance the right balance to get the desired to get the desired to get the desired result, not result, not result, not just code. just code. just code. Yes, of course. So, I'm Yes, of course. So, I'm Yes, of course. So, I'm doing this too. I doing this too. I doing this too. I created a tool created a tool created a tool that I call context or that I call context or that I call context or CTX. It started as a CTX. It started as a CTX. It started as a memory layer for memory layer for memory layer for agents, as I, agents, as I, agents, as I, like all of us, am actively like all of us, am actively like all of us, am actively experimenting with ChatGPT, experimenting with ChatGPT, experimenting with ChatGPT, Cloud Code, Pi, etc. And I Cloud Code, Pi, etc. And I Cloud Code, Pi, etc. And I wanted them all to be wanted them all to be wanted them all to be able to write and able to write and able to write and read data from read data from read data from memory, and that started memory, and that started memory, and that started working. And the working. And the working. And the more I ran more I ran more I ran multiple agents, and my multiple agents, and my multiple agents, and my typical day is, you typical day is, you typical day is, you know, somewhere between 18-30 know, somewhere between 18-30 know, somewhere between 18-30 tabs in the terminal, tabs in the terminal, tabs in the terminal, the more I realized the more I realized the more I realized that it would be an interesting that it would be an interesting that it would be an interesting experiment experiment experiment to try to to try to to try to centralize the centralize the centralize the credentials for credentials for credentials for verification in a verification in a verification in a web service. So, CTX web service. So, CTX is a simple webhost is a simple webhost is a simple webhost running on Cloudflare.

running on Cloudflare. running on Cloudflare. At first, it was written At first, it was written At first, it was written "by feel." It "by feel." It "by feel." It works. It started works. It started works. It started as a memory. He could as a memory. He could as a memory. He could give secrets to give secrets to give secrets to agents so they would agents so they would agents so they would n't see them and n't see them and n't see them and gain access to gain access to gain access to credentials. And that credentials. And that credentials. And that worked too. And worked too. And worked too. And now, as an additional now, as an additional now, as an additional experiment, I'm experiment, I'm experiment, I'm creating verifiers creating verifiers . It works like this: . It works like this: . It works like this: agents are assigned agents are assigned agents are assigned jobs, like we all jobs, like we all jobs, like we all do, through do, through do, through prompts. They go prompts. They go prompts. They go and do this work and do this work . They go back through the . They go back through the . They go back through the MCP connection to the CTX and MCP connection to the CTX and MCP connection to the CTX and say, "Here's what I say, "Here's what I say, "Here's what I finished. Did I finished. Did I finished. Did I do it? Please do it? Please do it? Please check it." check it." check it." Let's say, in Let's say, in Let's say, in the example of one of the the example of one of the the example of one of the workflows, workflows, workflows, you need to check you need to check you need to check whether the PR was created whether the PR was created whether the PR was created correctly. So, this correctly. So, this correctly. So, this will run the evaluator will run the evaluator will run the evaluator on a known on a known on a known verifier. To verifier. To verifier. To start, it could just be a start, it could just be a start, it could just be a simple CLI like simple CLI like simple CLI like in my repository. in my repository. in my repository. And then it becomes a And then it becomes a And then it becomes a reliable reliable reliable tool that tool that tool that can be can be can be used used used over and over again. It over and over again. It over and over again. It contacts the GitHub API contacts the GitHub API contacts the GitHub API and checks whether and checks whether and checks whether this PR actually exists, this PR actually exists, this PR actually exists, whether it is correct, and whether whether it is correct, and whether whether it is correct, and whether all merge checks are all merge checks are all merge checks are successful. And only successful. And only successful. And only then will it mark then will it mark then will it mark this node as this node as this node as completed in the completed in the completed in the workflow. Otherwise, workflow. Otherwise, workflow. Otherwise, it it it will redirect the agent will redirect the agent will redirect the agent and say, "Here are the things that and say, "Here are the things that and say, "Here are the things that still don't work." still don't work." still don't work." Right? So, Right? So, Right? So, again, it's an again, it's an again, it's an experiment, but it's a experiment, but it's a experiment, but it's a way that we've way that we've way that we've tried to tried to tried to experiment experiment experiment to see what to see what to see what patterns will catch on patterns will catch on patterns will catch on and what works and what doesn't.

and what works and what doesn't. and what works and what doesn't. But it's also just a But it's also just a But it's also just a fun time for fun time for fun time for this kind of "vibe coding," this kind of "vibe coding," this kind of "vibe coding," because I demand weird because I demand weird because I demand weird pixel art pixel art pixel art in all my in all my in all my apps. So you apps. So you apps. So you see, in this see, in this see, in this case, there was a simple case, there was a simple case, there was a simple workflow. And workflow. And workflow. And one of the one of the one of the verification verification verification steps requires that steps requires that steps requires that all merge requests all merge requests all merge requests be merged. This be merged. This be merged. This work is carried out by a work is carried out by a work is carried out by a centralized centralized centralized body. It body. It body. It uses its uses its uses its verifier, verifier, verifier, validates the API or validates the API or validates the API or other artifact, and other artifact, and other artifact, and then finally outputs a then finally outputs a then finally outputs a status in the status in the status in the sidebar on the right, if sidebar on the right, if sidebar on the right, if you even scroll you even scroll you even scroll down, with links down, with links down, with links to the verified to the verified to the verified repository and PR, right? repository and PR, right? repository and PR, right? So, a simple example So, a simple example : you can imagine : you can imagine : you can imagine standardized standardized standardized verifiers in verifiers in verifiers in a library, skills, a library, skills, a library, skills, or tools or tools or tools that you share within that you share within that you share within your organization. This your organization. This your organization. This could be one could be one could be one way to way to way to speed up speed up speed up PR work and gradually PR work and gradually PR work and gradually reduce the need for reduce the need for reduce the need for human oversight. human oversight. human oversight. Ugh. And there's a similar Ugh. And there's a similar Ugh. And there's a similar tool that I tool that I tool that I created, it's created, it's created, it's called Ryker. It's called Ryker. It's called Ryker. It's more of an engineering- more of an engineering- oriented Grokbot or oriented Grokbot or oriented Grokbot or Muse that tries Muse that tries Muse that tries to be as to be as to be as flexible as possible in how you flexible as possible in how you flexible as possible in how you hand it hand it hand it tasks, but once tasks, but once tasks, but once you do, you do, you do, its job is to its job is to follow through. follow through. follow through. So I don't want to So I don't want to So I don't want to constantly "babysit constantly "babysit " with all the PRs I'm " with all the PRs I'm completing. At a certain completing. At a certain point, point, my my work efficiency decreases, because I work efficiency decreases, because I work efficiency decreases, because I end up just " end up just " babysitting" PR: babysitting" PR: babysitting" PR: correcting tests, correcting tests, correcting tests, responding to feedback, responding to feedback, responding to feedback, comments, and all comments, and all comments, and all that. So I want to that. So I want to that. So I want to be able to be able to be able to lay out an idea of lay out an idea of lay out an idea of what we need to do and what we need to do and what we need to do and how to verify how to verify how to verify that that that work, and then work, and then work, and then hand it off to Ryker. He hand it off to Ryker. He hand it off to Ryker. He can take on the

can take on the can take on the job of checking job of checking job of checking that the reviews are done that the reviews are done that the reviews are done , that the , that the , that the fixed code meets fixed code meets fixed code meets our overall goal of our overall goal of our overall goal of producing working producing working software, and then software, and then just confirm just confirm just confirm it and let me know it and let me know it and let me know when I'm needed, when I'm needed, when I'm needed, instead of me having to keep track of instead of me having to keep track of instead of me having to keep track of all that. And all that. And all that. And with all of that, it just with all of that, it just with all of that, it just creates a test, creates a test, creates a test, determines which determines which determines which tools to tools to tools to run, runs run, runs run, runs them, makes sure them, makes sure them, makes sure everything went successfully, everything went successfully, everything went successfully, stores the evidence, stores the evidence, stores the evidence, shows it to me in shows it to me in shows it to me in different ways, and different ways, and different ways, and then decides when then decides when then decides when to stop. So to stop. So to stop. So if he can if he can if he can confirm, "yes, it confirm, "yes, it confirm, "yes, it worked, we worked, we worked, we fixed that bug fixed that bug ," for example, we ," for example, we ," for example, we put everything together put everything together put everything together and it works. This and it works. This and it works. This really takes the pressure off really takes the pressure off really takes the pressure off me from having to me from having to me from having to get bogged down in the minutiae of get bogged down in the minutiae of churning out the final 10%, churning out the final 10%, allowing me allowing me allowing me to focus on to focus on to focus on where I am valuable as a person: where I am valuable as a person: where I am valuable as a person: why we do this why we do this why we do this work and, to some work and, to some work and, to some extent, how we do it extent, how we do it . . Okay, maybe as a Okay, maybe as a thought thought thought experiment if experiment if experiment if you like, let's you like, let's you like, let's just raise our just raise our just raise our hands or exclaim: hands or exclaim: hands or exclaim: how did we developers how did we developers how did we developers usually test usually test usually test web applications before, web applications before, web applications before, before the LLM era?

before the LLM era? So, what would you do So, what would you do to check? to check? to check? Let's outline Let's outline Let's outline a few simple a few simple a few simple options to get options to get options to get you started. Well, you started. Well, you started. Well, for example, refresh for example, refresh for example, refresh the page, is the app the page, is the app the page, is the app still displayed, still displayed, still displayed, right? What exactly? right? What exactly? right? What exactly? Absolutely, yes. Absolutely, yes. Absolutely, yes. Open a new Open a new Open a new tab, check tab, check tab, check if everything works, right? if everything works, right? if everything works, right? What response codes do I What response codes do I What response codes do I get from the API? Are get from the API? Are get from the API? Are they expected given they expected given they expected given what I what I what I sent? Certainly. sent? Certainly. sent? Certainly. Local host or Local host or Local host or storage, storage, storage, data consistency checking, data consistency checking, data consistency checking, console logs, and console logs, and console logs, and so on. The network, uh, so on. The network, uh, so on. The network, uh, as you said. And, um, as you said. And, um, as you said. And, um, like, reading like, reading like, reading what's actually what's actually what's actually on the page. Is the content of on the page. Is the content of on the page. Is the content of the page actually... the page actually... the page actually... Interacting with it, right? Interacting with it, right? Interacting with it, right? And the reason we're And the reason we're And the reason we're talking about this is to talking about this is to talking about this is to prepare for the prepare for the prepare for the transition from a transition from a transition from a time when we, time when we, time when we, human developers, were human developers, were human developers, were constantly doing it constantly doing it constantly doing it ourselves. This was ourselves. This was ourselves. This was part of our part of our part of our core responsibilities core responsibilities core responsibilities when we were, so to speak when we were, so to speak , writing code by hand. , writing code by hand. , writing code by hand. Now, what's the difference Now, what's the difference Now, what's the difference between this and the early between this and the early between this and the early stages of the LLM era, when we stages of the LLM era, when we stages of the LLM era, when we say, "Here are three say, "Here are three say, "Here are three paragraphs of text. paragraphs of text. Follow these Follow these instructions." The difference instructions." The difference instructions." The difference between these things is the between these things is the between these things is the agent's ability agent's ability agent's ability to verify that it actually to verify that it actually to verify that it actually did what you did what you did what you asked it to do in the real asked it to do in the real asked it to do in the real application, and that application, and that application, and that its changes didn't break its changes didn't break its changes didn't break something else. Yes? Otherwise, something else. Yes? Otherwise, something else. Yes? Otherwise, you only get you only get you only get the hallucinations that LLMs the hallucinations that LLMs the hallucinations that LLMs naturally produce when naturally produce when trying trying to process the text.

to process the text. to process the text. They are trying to They are trying to They are trying to make you feel good. make you feel good. Uh, your agent can't Uh, your agent can't do any of do any of do any of this just in the this just in the this just in the terminal. Uh, he terminal. Uh, he terminal. Uh, he needs tools for that needs tools for that needs tools for that . Uh, . Uh, . Uh, fortunately, tools are fortunately, tools are fortunately, tools are now very easy now very easy now very easy to add to to add to to add to agents. agents. agents. Um, and we just need to Um, and we just need to Um, and we just need to teach them how teach them how teach them how to use them. We will to use them. We will to use them. We will tell them specifically tell them specifically tell them specifically what we are trying what we are trying what we are trying to prove. Um, when you to prove. Um, when you to prove. Um, when you do that, you don't do that, you don't do that, you don't want it to want it to want it to just be a paragraph just be a paragraph just be a paragraph like, "Hey, do this, like, "Hey, do this, like, "Hey, do this, always check always check always check changes in the browser." changes in the browser." changes in the browser." Whatever that means, it's Whatever that means, it's Whatever that means, it's pretty vague. pretty vague. pretty vague. Uh, "Don't forget to Uh, "Don't forget to Uh, "Don't forget to reboot and reboot and reboot and check." This is check." This is check." This is a little more specific. Er a little more specific. Er , "Check if , "Check if , "Check if data is saved." data is saved." data is saved." Check the console for Check the console for Check the console for errors. Well, to be errors. Well, to be more specific, it's a more specific, it's a large amount of data large amount of data large amount of data that can be put in there that can be put in there that can be put in there , and you , and you , and you can certainly do that. can certainly do that. can certainly do that. This is how This is how This is how many skills work. This is how many skills work. This is how many skills work. This is how many many many saved prompts work. saved prompts work. saved prompts work. But if you But if you But if you forget something or something forget something or something forget something or something changes, it leaves changes, it leaves changes, it leaves a lot of room for a lot of room for a lot of room for interpretation. interpretation. interpretation. It might be better to have a It might be better to have a It might be better to have a script that can script that can script that can check this in check this in check this in different ways. different ways. So you can So you can run it over run it over run it over and over again, doing a and over again, doing a and over again, doing a more deterministic more deterministic more deterministic check on a number of check on a number of check on a number of parameters, and then parameters, and then parameters, and then pass that pass that pass that result back to result back to result back to the agent so they the agent so they the agent so they know exactly what went wrong and know exactly what went wrong and know exactly what went wrong and how to fix it quickly how to fix it quickly how to fix it quickly . This closes . This closes . This closes the loop, which the loop, which the loop, which speeds up the process significantly: speeds up the process significantly: speeds up the process significantly: the agent can simply the agent can simply the agent can simply restart it, restart it, restart it, check, check, check, verify, and verify, and verify, and repeat until the repeat until the repeat until the errors disappear.

errors disappear. errors disappear. It's also unified, It's also unified, It's also unified, if you think about it, right if you think about it, right ? We know that LLMs are ? We know that LLMs are ? We know that LLMs are non-deterministic, non-deterministic, non-deterministic, so even if so even if so even if you give 30 agents you give 30 agents you give 30 agents the same clues, the the same clues, the the same clues, the outcome may outcome may outcome may vary. And vary. And vary. And reducing this to a reducing this to a reducing this to a single command that's single command that's single command that's based on based on based on real bash commands, real bash commands, real bash commands, bash calls, right bash calls, right ? This is much more ? This is much more ? This is much more deterministic. By deterministic. By deterministic. By the way, people like the way, people like the way, people like Lauren Tan and others who Lauren Tan and others who Lauren Tan and others who do thousands of PRs a do thousands of PRs a do thousands of PRs a month and build month and build month and build ambitious products that ambitious products that ambitious products that were previously were previously were previously impossible to build impossible to build impossible to build in that amount of time in that amount of time in that amount of time have mentioned that this is the have mentioned that this is the have mentioned that this is the first thing they first thing they first thing they focused on and it focused on and it focused on and it really unlocked really unlocked really unlocked speed. So, if speed. So, if speed. So, if your web application doesn't your web application doesn't your web application doesn't have a dedicated CLI have a dedicated CLI have a dedicated CLI that allows the agent to that allows the agent to that allows the agent to fully fully fully utilize the utilize the utilize the functionality you need functionality you need functionality you need to test, it to test, it to test, it will be much will be much will be much harder for you to work harder for you to work harder for you to work with just hints with just hints with just hints compared to compared to compared to being able to provide being able to provide being able to provide each agent with a each agent with a each agent with a tool to tool to tool to confirm confirm confirm that the feature is actually that the feature is actually that the feature is actually implemented. This is implemented. This is implemented. This is really effective. This did really effective. This did really effective. This did not cause any errors not cause any errors not cause any errors and the application looks and the application looks and the application looks correct. correct.

So for this So for this workshop we created a workshop we created a workshop we created a small tool small tool . We've run it . We've run it . We've run it a few times, it's a few times, it's a few times, it's called snack. You called snack. You called snack. You can simply can simply can simply run it run it run it using pnpm snack in the using pnpm snack in the using pnpm snack in the repository. It is not repository. It is not repository. It is not intended to be intended to be reusable, it is reusable, it is more of a template for more of a template for more of a template for how you can set something how you can set something how you can set something similar up for your similar up for your similar up for your applications to applications to applications to integrate this into your integrate this into your integrate this into your work with LLMs who work with LLMs who work with LLMs who are working on this are working on this are working on this application with application with application with you. So, snack can you. So, snack can you. So, snack can do several things. do several things. do several things. It can manage the It can manage the health status of health status of the application. He can the application. He can the application. He can start it and start it and start it and stop it. He can stop it. He can stop it. He can run a run a run a check command to check command to check command to see if the see if the see if the database is working. Is there a database is working. Is there a database is working. Is there a network connection network connection ? And so on. This is a quick ? And so on. This is a quick ? And so on. This is a quick check he check he check he can do. It also can do. It also can do. It also has a has a has a control function, so you can control function, so you can control function, so you can order it order it order it to join. to join. to join. Remember, this is our Remember, this is our Remember, this is our waiting list for a waiting list for a waiting list for a midnight snack. midnight snack. midnight snack. So, he can So, he can So, he can join the join the join the waiting list. waiting list. waiting list. He can He can He can reload reload reload the page, reset all the page, reset all the page, reset all data, only to data, only to data, only to find nothing. You can find nothing. You can find nothing. You can make it look for make it look for make it look for things: take a things: take a things: take a snapshot, browse snapshot, browse snapshot, browse network storage, network storage, network storage, console logs, console logs, console logs, local storage, local storage, local storage, take screenshots, take screenshots, take screenshots, run assessments— run assessments— all of these things at once. And all of these things at once. And all of these things at once. And ultimately, he can ultimately, he can ultimately, he can confirm this in two confirm this in two confirm this in two ways. It can ways. It can ways. It can execute a execute a execute a proof command, proof command, proof command, checking the checking the checking the deterministic deterministic deterministic result. Or a "judge result. Or a "judge " who uses the " who uses the " who uses the "system one "system one " model to understand whether " model to understand whether " model to understand whether we are moving in the we are moving in the we are moving in the right direction right direction , even if it is not , even if it is not , even if it is not one-to- one-to- one-to- one.

one. one. To fully To fully To fully develop this idea: develop this idea: develop this idea: when you do something when you do something when you do something like this in a like this in a like this in a repository where repository where repository where it hasn't been done before, the it hasn't been done before, the it hasn't been done before, the speed boost speed boost speed boost you get is you get is you get is n't just n't just n't just for you, right? This is for you, right? This is for you, right? This is for everyone else on for everyone else on for everyone else on your team who your team who your team who comes in later and comes in later and comes in later and can also can also can also generate significantly generate significantly generate significantly more PR much more PR much more PR much faster and more safely, faster and more safely, faster and more safely, right? It's right? It's right? It's safer for everyone on the safer for everyone on the safer for everyone on the team to actually team to actually team to actually benefit. And benefit. And benefit. And again, that's why again, that's why again, that's why we keep we keep we keep emphasizing that this is a emphasizing that this is a emphasizing that this is a fundamental fundamental fundamental concept. concept. concept. Yes. And if Yes. And if Yes. And if you think about it, that's you think about it, that's you think about it, that's really a huge really a huge really a huge part of it all. part of it all. part of it all. This may be even more This may be even more This may be even more important than your important than your software, than the software, than the application itself: the way application itself: the way application itself: the way you you you verify it, verify it, verify it, test it, and prove test it, and prove test it, and prove that it works, because that's what that it works, because that's what that it works, because that's what you can you can you can use, as use, as use, as many many many people do when switching to people do when switching to people do when switching to Rust. You can take Rust. You can take Rust. You can take your application, like the your application, like the your application, like the Bun team did, Bun team did, Bun team did, rewrite it in Rust rewrite it in Rust , Go, TypeScript, , Go, TypeScript, , Go, TypeScript, convert it to Go—not convert it to Go—not convert it to Go—not necessarily with the necessarily with the necessarily with the same level of LLM help, same level of LLM help, same level of LLM help, but they did but they did but they did use LLM use LLM use LLM to do it. And I think to do it. And I think to do it. And I think Theo Theo Theo rewrote it rewrote it rewrote it in Rust again last week, so.

in Rust again last week, so. in Rust again last week, so. Yes. Yes. Yes. Well, when you Well, when you Well, when you want it, if you have want it, if you have want it, if you have all these verifiers, all these verifiers, all these verifiers, then your source code does then your source code does then your source code does n't n't n't matter that much anymore. You matter that much anymore. You matter that much anymore. You can rewrite can rewrite can rewrite it and prove that it it and prove that it it and prove that it works. Continue. works. Continue. works. Continue. Certainly. Certainly. So, these tools So, these tools give your agents give your agents give your agents eyes. Or, if eyes. Or, if eyes. Or, if you think about it, you think about it, you think about it, the tools an the tools an the tools an agent uses agent uses agent uses give him a body. LLM give him a body. LLM is the brain, and the way is the brain, and the way is the brain, and the way it interacts it interacts it interacts physically, uh, physically, uh, physically, uh, virtually on your virtually on your virtually on your computer, is through computer, is through computer, is through these tools. And these tools. And these tools. And so, things like " so, things like " join" allow it join" allow it join" allow it to run and to run and to run and perform actions, and perform actions, and perform actions, and then check: did then check: did then check: did I somehow succeed in doing this I somehow succeed in doing this I somehow succeed in doing this ? ? Checking storage, Checking storage, starting a starting a starting a reboot, taking a reboot, taking a reboot, taking a snapshot, etc. So, snapshot, etc. So, snapshot, etc. So, let's run a let's run a let's run a few of them and few of them and few of them and see. P snack up. This see. P snack up. This see. P snack up. This opened our browser. opened our browser. And then I'll open another And then I'll open another browser or another browser or another browser or another terminal. I have terminal. I have terminal. I have too many terminals. too many terminals. And I'll run P snack snapshot. And And I'll run P snack snapshot. And this will give us a snapshot of this will give us a snapshot of this will give us a snapshot of the page in JSON format the page in JSON format that we can that we can that we can see. If I see. If I see. If I type in "Zack at zack.zack" and type in "Zack at zack.zack" and type in "Zack at zack.zack" and add that to the page.

add that to the page. Hopefully we'll Hopefully we'll see a few " see a few " Zacks" there. That's all. So, Zacks" there. That's all. So, Zacks" there. That's all. So, this is a way we this is a way we this is a way we can check. We can check. We can check. We can also take a can also take a can also take a screenshot to make it screenshot to make it screenshot to make it a little easier. This a little easier. This a little easier. This should open the should open the should open the screenshot. And so. This gave screenshot. And so. This gave screenshot. And so. This gave us the opportunity to not us the opportunity to not us the opportunity to not just pass it on just pass it on just pass it on to LLM for to LLM for to LLM for interpretation, but interpretation, but interpretation, but to bring it back to us. to bring it back to us. to bring it back to us. That's the ultimate goal, right That's the ultimate goal, right That's the ultimate goal, right ? Proving to a person ? Proving to a person . Ability to take . Ability to take . Ability to take screenshots, screenshots, screenshots, record videos, and record videos, and record videos, and add them to a add them to a add them to a pull request. You pull request. You pull request. You can do this can do this can do this using the GH CLI now. using the GH CLI now. Chrome debugging tools. Chrome debugging tools, yes . Where is the . Where is the . Where is the " " " bottleneck" actually located? Don't guess or bottleneck" actually located? Don't guess or bottleneck" actually located? Don't guess or waste time waste time waste time for 3 for 3 for 3 hours. Find out which hours. Find out which hours. Find out which part was part was part was causing the slow causing the slow causing the slow rendering. rendering. rendering. Yes. So the key point here is that it's Yes. So the key point here is that it's Yes. So the key point here is that it's not just the code not just the code not just the code you can you can you can look at that determines look at that determines look at that determines whether whether whether this application is working correctly; you this application is working correctly; you this application is working correctly; you can create can create can create tools that tools that tools that help prove to help prove to help prove to you that it works you that it works you that it works correctly without correctly without correctly without having to having to having to read the code at all. Because that's all read the code at all. Because that's all read the code at all. Because that's all we care about. Is this what the we care about. Is this what the user sees? Does user sees? Does it it it display correctly in display correctly in display correctly in this browser at this browser at this browser at this width, in this width, in this width, in dark theme, in dark theme, in dark theme, in light theme, etc.? You light theme, etc.? You light theme, etc.? You can run can run can run endless tests on endless tests on endless tests on this over and over in a this over and over in a this over and over in a loop, which, as loop, which, as loop, which, as Zach said, allows your Zach said, allows your Zach said, allows your team to move team to move team to move faster because you've faster because you've faster because you've given the agent given the agent given the agent the ability to the ability to the ability to run these run these run these checks themselves. So if

checks themselves. So if checks themselves. So if we run this, we run this, we run this, for example, run for example, run for example, run doctor, add [email protected], doctor, add [email protected], doctor, add [email protected], and then check the and then check the and then check the repository—it's there. repository—it's there. repository—it's there. If we check If we check If we check the repository, it's the repository, it's the repository, it's not there. We will see that the not there. We will see that the not there. We will see that the local storage is local storage is local storage is empty. And if we empty. And if we empty. And if we reboot and reboot and reboot and take a snapshot, we'll take a snapshot, we'll take a snapshot, we'll see that see that see that it's really empty. it's really empty. it's really empty. This is a way for the This is a way for the This is a way for the tool to tool to tool to work in a loop, work in a loop, work in a loop, checking all these checking all these checking all these things and yes, things and yes, things and yes, validating them validating them validating them before they even before they even before they even get back to us. Oh, get back to us. Oh, get back to us. Oh, I see we I see we I see we don't actually don't actually don't actually keep it. I forgot to keep it. I forgot to keep it. I forgot to do that. And the agent do that. And the agent do that. And the agent can just move on can just move on can just move on . Quick, we . Quick, we . Quick, we wanted to stop and wanted to stop and wanted to stop and answer a few answer a few answer a few questions, if there are any. questions, if there are any. questions, if there are any. Yes, sir. Yes, sir. Yes, sir. Yes. The question is, Yes. The question is, Yes. The question is, do we use do we use do we use models to write models to write models to write JSON to then JSON to then JSON to then help make help make help make our code our code our code deterministic and deterministic and deterministic and bring that back? bring that back? The answer for me is yes. I use yes. I use yes. I use this, run it, and then this, run it, and then this, run it, and then apply unit apply unit apply unit tests to test tests to test tests to test it in different ways, it in different ways, it in different ways, find edge find edge find edge cases, and cases, and cases, and make sure make sure make sure the code is a true the code is a true the code is a true interpretation of what interpretation of what interpretation of what I expect to get I expect to get I expect to get from the first- from the first- from the first- level model, and use level model, and use level model, and use this to prove that I'm this to prove that I'm this to prove that I'm on the right track.

on the right track. But I also keep But I also keep them as an extra them as an extra them as an extra check to check to check to make sure make sure make sure nothing nothing nothing changes or changes or changes or deviates over time. The deviates over time. The deviates over time. The general process general process general process we are talking about? Yes, what we are talking about? Yes, what we are talking about? Yes, what I'm talking about, I'm talking about, I'm talking about, at least on our at least on our at least on our team: we build team: we build team: we build a lot of internal a lot of internal a lot of internal tools. We tools. We tools. We also have several also have several also have several client applications client applications . It's usually TypeScript, and . It's usually TypeScript, and . It's usually TypeScript, and they deploy they deploy they deploy to Cloudflare. So, these are to Cloudflare. So, these are to Cloudflare. So, these are complete full-stack complete full-stack applications. This works applications. This works applications. This works perfectly. You perfectly. You perfectly. You can even conduct can even conduct can even conduct assessments focused assessments focused assessments focused on the backend or on the backend or on the backend or database operations database operations database operations . You can be . You can be . You can be as as as detailed detailed detailed as you want. But as you want. But as you want. But it's absolutely certain, it's absolutely certain, it's absolutely certain, yes, it works on a yes, it works on a yes, it works on a full stack. I full stack. I full stack. I expect that in expect that in expect that in the future we the future we the future we will interact with will interact with will interact with software in a software in a similar way to similar way to gardeners gardeners gardeners tending their tending their tending their garden so that it garden so that it garden so that it grows properly grows properly . . . Yes. Independently. Yes. Yes. Independently. Yes. Yes. Independently. Yes. That's why we don't That's why we don't That's why we don't provide a ready-made provide a ready-made provide a ready-made tool, because right tool, because right tool, because right now we can't now we can't now we can't offer a offer a offer a universal tool universal tool universal tool that would be suitable for that would be suitable for that would be suitable for every application every application every application you work on here. you work on here. you work on here. You need to You need to You need to create your own create your own create your own tools that tools that tools that fit your fit your fit your application, then application, then application, then hand them over to your hand them over to your hand them over to your models and make sure models and make sure models and make sure they know how they know how they know how to use them well.

to use them well. to use them well. Um, because Um, because Um, because I also wanted to say I also wanted to say that your that your that your evaluation criteria, what's evaluation criteria, what's evaluation criteria, what's really important really important really important to your team or your to your team or your to your team or your business, will business, will business, will be significantly different be significantly different be significantly different depending on whether depending on whether depending on whether you work you work you work in, say, in, say, in, say, interior design or interior design or interior design or where speed is the where speed is the where speed is the only thing that only thing that only thing that matters. Um, I think matters. Um, I think matters. Um, I think your your your evaluation criteria will be very evaluation criteria will be very evaluation criteria will be very different, and it makes sense different, and it makes sense different, and it makes sense to develop them together to develop them together to develop them together with your team. with your team. with your team. Yes. There are many Yes. There are many Yes. There are many ways to pass ways to pass ways to pass data to a model so that data to a model so that data to a model so that it can it can it can interpret it and interpret it and interpret it and keep up with the process. keep up with the process. This is an important part of This is an important part of keeping keeping keeping the model up to date. The the model up to date. The the model up to date. The AMP team has a great AMP team has a great AMP team has a great article about this. I don't article about this. I don't article about this. I don't remember the title of remember the title of remember the title of the post, but I'll the post, but I'll the post, but I'll definitely find it definitely find it . I think it's . I think it's . I think it's called " called " cycle-based development", I'm not cycle-based development", I'm not sure. I can't sure. I can't sure. I can't remember. But remember. But remember. But their example their example their example shows a canvas with a shows a canvas with a shows a canvas with a circle and a ball circle and a ball circle and a ball bouncing inside bouncing inside bouncing inside it, and there's a it, and there's a it, and there's a small hole small hole small hole through which the ball can through which the ball can through which the ball can fall out under certain fall out under certain fall out under certain conditions, after which the game conditions, after which the game conditions, after which the game ends. And the whole ends. And the whole ends. And the whole idea is to idea is to idea is to make things make things feedback-friendly. That is the feedback-friendly. That is the main thing.

main thing. main thing. When the model doesn't see When the model doesn't see When the model doesn't see the canvas in real the canvas in real the canvas in real time, you can, as a time, you can, as a time, you can, as a very simple example, very simple example, very simple example, output the coordinates of output the coordinates of output the coordinates of the ball at each the ball at each the ball at each stage, and you will notice stage, and you will notice stage, and you will notice an error: when it an error: when it an error: when it reaches a certain point, reaches a certain point, reaches a certain point, it jumps to another it jumps to another it jumps to another part of the screen. If part of the screen. If part of the screen. If it moves more it moves more it moves more than one pixel than one pixel than one pixel at a time, in one cycle, at a time, in one cycle, at a time, in one cycle, then something is probably then something is probably then something is probably wrong. So, there is wrong. So, there is wrong. So, there is an error there. If you an error there. If you an error there. If you can provide this to the can provide this to the can provide this to the model, it will help model, it will help model, it will help you with debugging, you with debugging, you with debugging, because it doesn't have eyes because it doesn't have eyes because it doesn't have eyes to see the canvas and to see the canvas and to see the canvas and notice that the ball is notice that the ball is notice that the ball is behaving behaving behaving incorrectly, incorrectly, incorrectly, for example with physics. for example with physics. for example with physics. So, So, So, And also, once you And also, once you And also, once you do that, you do that, you do that, you can create 30 can create 30 can create 30 different different software options that software options that work with that work with that work with that assessment and choose the one assessment and choose the one assessment and choose the one you you you need. need. need. Yes. Yes. Yes. Um, that's another thing Um, that's another thing Um, that's another thing this reveals, right? this reveals, right? this reveals, right? This is really This is really This is really fundamental to fundamental to fundamental to development, and especially development, and especially development, and especially if we expect if we expect if we expect agents and models agents and models agents and models to improve to improve , and tokens to possibly , and tokens to possibly , and tokens to possibly become cheaper, then you become cheaper, then you become cheaper, then you can imagine us can imagine us can imagine us all coming in on all coming in on all coming in on Monday and saying, “ Monday and saying, “ I want to see 35 I want to see 35 I want to see 35 versions of this new versions of this new versions of this new feature or application.” feature or application.” feature or application.” Of course. Of course. Of course. And choose one.

And choose one. And choose one. So how do we actually So how do we actually So how do we actually launch or launch or launch or create this? Well, in create this? Well, in create this? Well, in this case, as we this case, as we this case, as we said, we can't said, we can't said, we can't prove it for prove it for prove it for everything, universally. everything, universally. everything, universally. This should be something This should be something This should be something unique to your unique to your unique to your application. So for application. So for application. So for our “ our “ midnight snack midnight snack ” page, the validation command ” page, the validation command ” page, the validation command might look like this might look like this might look like this , where we essentially , where we essentially , where we essentially use a use a use a tool like tool like tool like Playwright to Playwright to Playwright to confirm that the email confirm that the email confirm that the email was successfully was successfully was successfully saved after saved after saved after being added to the being added to the being added to the waitlist. So, we waitlist. So, we waitlist. So, we can check that there can check that there are no email addresses on the waiting list. Then we addresses on the waiting list. Then we addresses on the waiting list. Then we enter the email on the enter the email on the enter the email on the next line, next line, next line, get a button and get a button and get a button and click it—the click it—the join join join waitlist button. And waitlist button. And waitlist button. And then we can then we can then we can expect the expect the expect the status element on the page status element on the page status element on the page to contain a to contain a to contain a waiting entry, and waiting entry, and waiting entry, and check the check the check the inner text to inner text to inner text to see if see if see if our email address is there. This is our email address is there. This is our email address is there. This is one of the simple one of the simple one of the simple ways, very easy ways, very easy if you've ever if you've ever if you've ever written end-to-end written end-to-end written end-to-end tests using tests using tests using Playwright or the Playwright or the Playwright or the React testing library, or React testing library, or React testing library, or something similar. You've something similar. You've something similar. You've definitely definitely definitely used used used similar tools before.

similar tools before. similar tools before. You can simply You can simply You can simply hand them back to hand them back to hand them back to the model or your the model or your the model or your agent and they will agent and they will agent and they will run them. Provide this run them. Provide this run them. Provide this as a tool and they as a tool and they as a tool and they can can can verify everything themselves, verify everything themselves, verify everything themselves, creating creating creating reproducible proof reproducible proof reproducible proof that can then that can then that can then be used. be used. be used. The next time The next time The next time the page changes, the page changes, the page changes, you can you can you can run this again and run this again and run this again and make sure make sure make sure nothing broke. nothing broke. nothing broke. This works as a regression This works as a regression , uh, and allows everyone , uh, and allows everyone , uh, and allows everyone on the team to move on the team to move on the team to move on. One on. One on. One caveat, caveat, caveat, a little advice: a a little advice: a a little advice: a successful successful successful screenshot is not screenshot is not screenshot is not proof. Uh, proof. Uh, proof. Uh, make sure it make sure it make sure it actually shows the actually shows the actually shows the state you state you state you expect it to, that it expect it to, that it expect it to, that it actually executed the actually executed the actually executed the commands to reach commands to reach commands to reach that state; because that state; because that state; because otherwise it could otherwise it could otherwise it could just be another just be another just be another hoax, and you see hoax, and you see hoax, and you see the picture in PR and think, " the picture in PR and think, " Oh yeah, that's, that's Oh yeah, that's, that's Oh yeah, that's, that's right." "But it right." "But it right." "But it could have caused other could have caused other could have caused other side effects. It side effects. It side effects. It could have gone down a different path could have gone down a different path that's that's that's not really possible within the not really possible within the not really possible within the interface. Uh, so interface. Uh, so interface. Uh, so you want to you want to you want to make sure that it make sure that it make sure that it actually runs actually runs actually runs from start to finish from start to finish from start to finish to test that. to test that.

Uh, yeah. And once Uh, yeah. And once you build these kinds of you build these kinds of you build these kinds of loops that are, you know, loops that are, you know, loops that are, you know, robust, reliable, robust, reliable, robust, reliable, repeatable, and repeatable, and repeatable, and uniform, the uniform, the uniform, the value or the main value or the main value or the main benefit is that benefit is that benefit is that the error the error the error starts to drive starts to drive starts to drive future work, future work, future work, right? So if in right? So if in right? So if in one loop the agent one loop the agent one loop the agent fails, uh, and fails, uh, and fails, uh, and you're busy talking to you're busy talking to you're busy talking to someone at work or someone at work or someone at work or starting another starting another starting another task, that agent task, that agent task, that agent can determine on its own, can determine on its own, can determine on its own, "That's not "That's not "That's not quite right."" quite right."" quite right."" Let me Let me Let me try again." try again." try again." Let me Let me Let me run the " run the " run the " check" command again. "Okay, check" command again. "Okay, check" command again. "Okay, now everything is good." now everything is good." now everything is good." When you think about what that When you think about what that When you think about what that means at scale, means at scale, means at scale, where a single PR or where a single PR or where a single PR or agent, a single agent, a single agent, a single workflow if workflow if workflow if you will, is you will, is you will, is self-healing, self-healing, self-healing, self-correcting, and figuring things out on its self-correcting, and figuring things out on its own, own, you don't have to you don't have to you don't have to go back to that go back to that go back to that PR at the end of the day and PR at the end of the day and PR at the end of the day and think, 'Oh, I still have to think, 'Oh, I still have to think, 'Oh, I still have to finish this finish this finish this .'" And if you just .'" And if you just .'" And if you just multiply that, right? "This, multiply that, right? "This, multiply that, right? "This, this is actually where, uh this is actually where, uh , again, we're , again, we're , again, we're just saying that this is the just saying that this is the just saying that this is the foundation for speed foundation for speed foundation for speed and performance. and performance. and performance. Yeah. Yeah. Yeah. And how many of you And how many of you And how many of you write good write good error messages? Raise your error messages? Raise your hands. hands. hands. Yeah, nobody knows. We Yeah, nobody knows. We Yeah, nobody knows. We already know. already know. already know. Sir, what's your Sir, what's your Sir, what's your message? message? message? I have small hands. I have small hands. I have small hands. There you have it.

There you have it. There you have it. Yeah. Yeah. Yeah. Great. Great. Great. Uh, well, you should. Uh Uh, well, you should. Uh , I think they're more , I think they're more , I think they're more important than important than ever, because when your ever, because when your ever, because when your application throws an application throws an application throws an error, guess error, guess error, guess who can who can who can interpret that interpret that interpret that error very well and act on error very well and act on error very well and act on it? The agent. it? The agent. it? The agent. So, uh, if you So, uh, if you So, uh, if you make your make your make your error clearly error clearly error clearly explain why it explain why it explain why it occurred and provide occurred and provide occurred and provide as much as much as much detail as possible about how detail as possible about how detail as possible about how that state was reached that state was reached that state was reached , then your agent , then your agent , then your agent can just can just can just self-heal. self-heal. self-heal. So, uh, it's very So, uh, it's very So, uh, it's very easy and very worthwhile to easy and very worthwhile to easy and very worthwhile to do that. do that. Yeah, and that's the concept Yeah, and that's the concept that Nick and I that Nick and I that Nick and I talked about, you know? And it talked about, you know? And it talked about, you know? And it sounds like we're, sounds like we're, sounds like we're, well, in in a sense well, in in a sense well, in in a sense we're trying to give we're trying to give we're trying to give LLM determinism, but LLM determinism, but LLM determinism, but we don't really we don't really we don't really want to. And that's also in want to. And that's also in want to. And that's also in the spirit of the spirit of the spirit of what the what the what the AMP team is talking about. Agents are AMP team is talking about. Agents are AMP team is talking about. Agents are intelligent. They're intelligent. They're intelligent. They're capable of a lot of things. capable of a lot of things. capable of a lot of things. In fact, we want In fact, we want In fact, we want them to be them to be them to be creative. Because creative. Because creative. Because often, if they have often, if they have often, if they have the right constraints, the right constraints, the right constraints, and constraints and constraints and constraints can often equal can often equal can often equal freedom, right? In freedom, right? In freedom, right? In a sense. Then a sense. Then a sense. Then they'll find creative they'll find creative they'll find creative solutions and actually solutions and actually solutions and actually solve what solve what solve what you asked them to do. That's why you asked them to do. That's why you asked them to do. That's why you give them a result you give them a result . That's why Nick's project that . That's why Nick's project that he's he's he's experimenting with is a experimenting with is a result-oriented design. And it's result-oriented design. And it's not about being not about being not about being too demanding about too demanding about too demanding about how the agent how the agent how the agent gets to the goal. That gets to the goal. That can actually can actually lead to lead to lead to reduced reduced reduced performance in some cases.

performance in some cases. performance in some cases. It's more about It's more about It's more about giving it a map and giving it a map and giving it a map and saying, you have to saying, you have to saying, you have to get here. You'll get here. You'll get here. You'll know you've got know you've got know you've got it when this it when this it when this command returns command returns command returns this. this. this. Yeah. Although you don't Yeah. Although you don't Yeah. Although you don't necessarily have to necessarily have to necessarily have to dictate exactly how dictate exactly how dictate exactly how it gets there. And it gets there. And it gets there. And that's becoming more and that's becoming more and that's becoming more and more true more true more true every day, especially every day, especially every day, especially with the advanced with the advanced with the advanced models. Fable, Astra, Saul models. Fable, Astra, Saul models. Fable, Astra, Saul 6.1, Opus 5.5. They're all very 6.1, Opus 5.5. They're all very 6.1, Opus 5.5. They're all very good at being, well, good at being, well, good at being, well, creative—that's creative—that's creative—that's how I would describe it. But how I would describe it. But how I would describe it. But if you give them a goal of, " if you give them a goal of, " if you give them a goal of, " This is where I want to get This is where I want to get This is where I want to get to," and don't to," and don't to," and don't necessarily necessarily necessarily give them a path, give them a path, give them a path, they can they can they can explore explore explore options and cover options and cover options and cover more edge more edge more edge cases. They can cases. They can cases. They can figure something out or figure something out or figure something out or ask you ask you ask you a question that will a question that will a question that will reveal something. But if you reveal something. But if you reveal something. But if you just say, "You just say, "You just say, "You need to get here, need to get here, need to get here, then here, then then here, then then here, then here," they can here," they can here," they can stick with stick with stick with it. And you want it. And you want it. And you want them to deviate a little bit them to deviate a little bit them to deviate a little bit , but , but , but get to the right get to the right get to the right end goal. And as I end goal. And as I end goal. And as I said, that's something we've said, that's something we've only really started to appreciate in the last only really started to appreciate in the last batch of advanced batch of advanced batch of advanced models. Where that models. Where that models. Where that really becomes really becomes really becomes possible. So it's possible. So it's possible. So it's only going to only going to only going to get better. get better. get better. Sure. Sure. So as the process goes, So as the process goes, you want to get into a you want to get into a you want to get into a typical TDD cycle, like typical TDD cycle, like typical TDD cycle, like red-green- red-green- refactoring with your refactoring with your refactoring with your agent. We've been saying, agent. We've been saying, agent. We've been saying, you've heard about the loops at you've heard about the loops at you've heard about the loops at each of these each of these each of these AIE events, and it's the same here, AIE events, and it's the same here, AIE events, and it's the same here, but you can but you can but you can use it use it use it directly in the directly in the directly in the verification process.

verification process. verification process. Test it, Test it, Test it, reproduce it, show reproduce it, show reproduce it, show me, fix it, and me, fix it, and me, fix it, and then provide proof of then provide proof of then provide proof of it. And how are you it. And how are you it. And how are you going to show going to show going to show me the proof? That's just as me the proof? That's just as me the proof? That's just as important as important as important as the fix itself, because that's what the fix itself, because that's what the fix itself, because that's what gives you gives you gives you peace of mind and allows you peace of mind and allows you peace of mind and allows you to trust the model that to trust the model that to trust the model that it's actually going to do the it's actually going to do the it's actually going to do the work. And then when work. And then when work. And then when it says, "Hey, I'm it says, "Hey, I'm it says, "Hey, I'm done," you know it's done," you know it's done," you know it's done because done because done because it's somehow it's somehow it's somehow proven it to you. proven it to you. Absolutely. So in this So in this example, you can example, you can example, you can run Snack Doctor. run Snack Doctor. run Snack Doctor. Let's just Let's just run it quickly. P Snack Doctor. So it runs, So it runs, and it shows us and it shows us and it shows us a bunch of checks that a bunch of checks that a bunch of checks that it's done. It it's done. It it's done. It checks if we have checks if we have checks if we have node, if Chromium is installed node, if Chromium is installed node, if Chromium is installed , if our , if our , if our session exists, if it's actually session exists, if it's actually session exists, if it's actually working, and if it was successfully working, and if it was successfully working, and if it was successfully built. That's what built. That's what built. That's what we run as a we run as a we run as a quick quick quick health check. Just a bunch of health check. Just a bunch of health check. Just a bunch of different things. Then we different things. Then we different things. Then we run the following run the following run the following command: Snack Join Ada at command: Snack Join Ada at command: Snack Join Ada at example.test.

example.test. Sorry, we're not Sorry, we're not printing anymore. printing anymore. Okay. Gave us Okay. Gave us the result. Snack storage. The the result. Snack storage. The storage is empty. He storage is empty. He didn't actually add didn't actually add didn't actually add Ada there. Ada there. Ada there. So this is the point So this is the point So this is the point where in the real where in the real where in the real world your agent—you've world your agent—you've world your agent—you've just just just benefited from this whole benefited from this whole benefited from this whole system, because system, because system, because at this point your at this point your at this point your agent understands: I'm not agent understands: I'm not agent understands: I'm not done yet and I'm not done yet and I'm not done yet and I'm not going to give up on this going to give up on this going to give up on this work halfway through. work halfway through. work halfway through. Yeah. And if you Yeah. And if you Yeah. And if you reboot, reboot, reboot, do the same thing do the same thing do the same thing again. Boom. We've now again. Boom. We've now again. Boom. We've now given our application given our application given our application "eyes." It can "eyes." It can "eyes." It can see this. It can see this. It can see this. It can run this. It run this. It run this. It can understand and can understand and can understand and change the world change the world change the world of the application, the midnight of the application, the midnight of the application, the midnight snack application, see snack application, see snack application, see the results of this, and the results of this, and the results of this, and then interpret then interpret what happened next. So what happened next. So what happened next. So you can try you can try you can try this. If we this. If we this. If we apply the apply the storage fix and add an storage fix and add an honest copy, then we honest copy, then we honest copy, then we can run can run can run the test. Let's the test. Let's the test. Let's run this. It run this. It run this. It ran the browser, and ran the browser, and ran the browser, and the test failed. I the test failed. I the test failed. I didn't actually didn't actually didn't actually run the test, but run the test, but run the test, but if we applied the if we applied the if we applied the fix. If we fix. If we fix. If we applied applied applied the fix, it would the fix, it would the fix, it would show that.

show that. Okay, one of the things Okay, one of the things that we've been that we've been that we've been thinking about a lot lately, thinking about a lot lately, thinking about a lot lately, talking about, and starting to talking about, and starting to talking about, and starting to implement in implement in implement in more applications is more applications is more applications is the concept of "checks." Well the concept of "checks." Well , you can take a , you can take a support ticket system where there's a certain support ticket system where there's a certain performance, performance, performance, how a person responds how a person responds how a person responds to a ticket and they consider that to be to a ticket and they consider that to be to a ticket and they consider that to be quality, excellent quality, excellent customer service. And then what does it customer service. And then what does it mean, well, mean, well, mean, well, to maintain that in to maintain that in to maintain that in some way, some way, some way, like, right? Um, like, right? Um, like, right? Um, why would we do that? why would we do that? why would we do that? There's a bunch of reasons, and we'll There's a bunch of reasons, and we'll There's a bunch of reasons, and we'll go over some of go over some of go over some of them. But, again, them. But, again, them. But, again, if you think about it, if if you think about it, if if you think about it, if we're all, uh, or if it we're all, uh, or if it we're all, uh, or if it seems like a lot seems like a lot seems like a lot of us are moving toward of us are moving toward of us are moving toward running running running RSI tools and RSI tools and self-evolving codebases, then self-evolving codebases, then everything that we do everything that we do everything that we do today, everything that today, everything that today, everything that your agent does your agent does your agent does today, every today, every today, every run throughout the day, run throughout the day, is very valuable data. It's is very valuable data. It's is very valuable data. It's very valuable data right very valuable data right very valuable data right now, because you now, because you now, because you can feed it into can feed it into can feed it into LLM and get the LLM and get the LLM and get the necessary necessary necessary fixes. It's fixes. It's fixes. It's incredibly valuable incredibly valuable incredibly valuable data for the future, data for the future, data for the future, as as as the tools the tools the tools improve and it becomes improve and it becomes improve and it becomes easier easier easier to tune to tune to tune RSI cycles, because then you RSI cycles, because then you RSI cycles, because then you have a data set of have a data set of have a data set of how exactly how exactly how exactly your system performed, for your system performed, for your system performed, for which you can which you can which you can create, uh, very create, uh, very create, uh, very complex assessments and complex assessments and complex assessments and let the agents let the agents let the agents climb the climb the climb the hill, which is to hill, which is to hill, which is to cyclically test cyclically test cyclically test their performance their performance their performance and keep doing and keep doing and keep doing it until they it until they it until they get get get better and better, better and better, better and better, right?

right? right? Yeah. And at that point Yeah. And at that point Yeah. And at that point they are able to they are able to they are able to improve improve improve themselves themselves themselves using this using this using this evidence from previous evidence from previous evidence from previous runs. So now, runs. So now, runs. So now, uh, if we fix uh, if we fix uh, if we fix this code, it now this code, it now this code, it now persists, uh, but persists, uh, but persists, uh, but it can also it can also it can also lie. So, after lie. So, after lie. So, after reloading the reloading the reloading the page it might page it might page it might say " say " say " waitlisted," uh waitlisted," uh , but the screenshot , but the screenshot , but the screenshot might actually might actually might actually say "you are on the list say "you are on the list ," uh, "your seat ," uh, "your seat ," uh, "your seat is confirmed." They is confirmed." They is confirmed." They don't exactly match up, don't exactly match up, don't exactly match up, so it could be so it could be so it could be a bug because a bug because a bug because they don't match they don't match they don't match each other. So we're each other. So we're each other. So we're kind of moving away from the kind of moving away from the kind of moving away from the static static static testing element. Like how do testing element. Like how do testing element. Like how do we prove that, uh, we prove that, uh, we prove that, uh, it worked without it worked without it worked without being very rigidly being very rigidly being very rigidly tied to the tied to the tied to the actual text on actual text on actual text on the screen, uh, for very a the screen, uh, for very a the screen, uh, for very a contrived example. contrived example. Uh, and so if we Uh, and so if we just run, just run, just run, you know, P snack, judge, we should you know, P snack, judge, we should you know, P snack, judge, we should run this, and it run this, and it run this, and it throws an error, uh, throws an error, uh, throws an error, uh, because even though everything because even though everything because even though everything worked, uh, and it worked, uh, and it worked, uh, and it says we're in says we're in says we're in the system, uh, Jev did the system, uh, Jev did the system, uh, Jev did n't see it, or n't see it, or n't see it, or the model didn't the model didn't the model didn't notice it. Uh, and that's notice it. Uh, and that's notice it. Uh, and that's because the confirmation text because the confirmation text violates the contract. It's violates the contract. It's not exactly one-to- not exactly one-to- not exactly one-to- one. So how do we one. So how do we one. So how do we fix that without adding fix that without adding fix that without adding a lot of hard- a lot of hard- a lot of hard- wired things to everything wired things to everything wired things to everything we do. The answer we do. The answer is the system is the system is the system one models, which we're going one models, which we're going one models, which we're going to talk about now.

to talk about now. to talk about now. Yeah, absolutely. So to Yeah, absolutely. So to Yeah, absolutely. So to put it in perspective put it in perspective put it in perspective for those of you who've never for those of you who've never for those of you who've never heard of it before, heard of it before, heard of it before, you know, the concept of you know, the concept of you know, the concept of system one and system one and system one and system two system two system two actually comes actually comes actually comes from psychology. It's a from psychology. It's a from psychology. It's a useful model for useful model for useful model for thinking about how the thinking about how the thinking about how the human human human brain works and how it brain works and how it brain works and how it thinks. So thinks. So thinks. So system one is system one is system one is super-fast decision-making. super-fast decision-making. super-fast decision-making. It's, well, it's It's, well, it's It's, well, it's autonomous. Like someone autonomous. Like someone autonomous. Like someone cut me off while I was cut me off while I was cut me off while I was driving, and I just driving, and I just driving, and I just reacted, not even reacted, not even reacted, not even thinking, thinking, thinking, you know? Or I almost you know? Or I almost you know? Or I almost lost my balance lost my balance lost my balance and, well, I leveled out. And and, well, I leveled out. And and, well, I leveled out. And system two is like, system two is like, system two is like, "I need to decide "I need to decide "I need to decide what to do with my what to do with my what to do with my life." So I'll life." So I'll life." So I'll book an Airbnb and I'll book an Airbnb and I'll book an Airbnb and I'll think about it for three days. think about it for three days. think about it for three days. And now, you know, if And now, you know, if And now, you know, if you apply that to the you apply that to the you apply that to the LLM world, now LLM world, now LLM world, now we have Jev, and right we have Jev, and right we have Jev, and right after that, after that, after that, two days later, two days later, two days later, Cloudflare's Klev. It's even Cloudflare's Klev. It's even Cloudflare's Klev. It's even faster. And this, this is a faster. And this, this is a faster. And this, this is a really exciting really exciting really exciting time to think about time to think about time to think about all the ways that all the ways that all the ways that these these these tools can be combined to tools can be combined to tools can be combined to create new create new create new primitives that we're going to primitives that we're going to primitives that we're going to continue to continue to continue to see in the see in the evolving AI architectures. evolving AI architectures. Yeah. Let's put it another way, Yeah. Let's put it another way, Yeah. Let's put it another way, that light over there, if that light over there, if that light over there, if I were to ask, "What I were to ask, "What I were to ask, "What color is it glowing? color is it glowing? " You'd quickly " You'd quickly " You'd quickly say, "Yellow." say, "Yellow." say, "Yellow." That's like That's like That's like system one thinking, right? You system one thinking, right? You system one thinking, right? You just know the color. just know the color. just know the color. And if I were to ask, " And if I were to ask, " Why is it glowing Why is it glowing Why is it glowing yellow?" "Well, because...

yellow?" "Well, because... " That's where we " That's where we " That's where we start to think. start to think. start to think. Now we're Now we're Now we're thinking, you know, thinking, you know, thinking, you know, we could we could we could go into physics. go into physics. go into physics. why the light why the light why the light is yellow is yellow is yellow . . . Why are we here? Why are we here? Why are we here? That's how we got to what we That's how we got to what we That's how we got to what we really really really want to talk to want to talk to want to talk to you about today. So you about today. So you about today. So here we have another here we have another here we have another game that we can game that we can game that we can play within this play within this . It's a quick snack game. . It's a quick snack game. . It's a quick snack game. We have a saboteur and a We have a saboteur and a We have a saboteur and a detective. So in this detective. So in this detective. So in this game, the saboteur will choose a game, the saboteur will choose a game, the saboteur will choose a snack. He'll put a snack. He'll put a snack. He'll put a secret bug in there, secret bug in there, secret bug in there, randomly randomly randomly inserting a bug. inserting a bug. inserting a bug. We don't know which one. It'll We don't know which one. It'll We don't know which one. It'll just be a just be a just be a coincidence. And then coincidence. And then coincidence. And then you, the detective, you, the detective, you, the detective, find it find it find it with the snack itself with the snack itself with the snack itself and prove what it is and prove what it is . Name the bug and . Name the bug and . Name the bug and provide evidence that you provide evidence that you provide evidence that you found it. So this is what found it. So this is what found it. So this is what you can you can you can do, if you've do, if you've do, if you've uploaded a site or a uploaded a site or a uploaded a site or a repository, you repository, you repository, you can just can just can just run pnpm snack up--bug run pnpm snack up--bug run pnpm snack up--bug random and let it random and let it random and let it run to run to run to try to find try to find try to find it. So let's it. So let's it. So let's try it. Let's give it try it. Let's give it try it. Let's give it some time. Uh, what some time. Uh, what some time. Uh, what was there?--bug.

was there?--bug. was there?--bug. Bug random. Bug random. Oh, it's already running. Oh, it's already running. Let me Let me Let me stop this. Stop the stop this. Stop the stop this. Stop the right process. right process. Okay. So here it's Okay. So here it's up and running, it has the up and running, it has the up and running, it has the app URL, it's running, it's not in app URL, it's running, it's not in app URL, it's running, it's not in headless mode, headless mode, headless mode, so we can so we can so we can see it. The bug, uh, see it. The bug, uh, see it. The bug, uh, it's it's it's hidden until you hidden until you hidden until you stop up. Yeah, stop up. Yeah, stop up. Yeah, right. Um, leave it right. Um, leave it right. Um, leave it running. In another running. In another running. In another terminal, run terminal, run terminal, run pnpm snack doctor. So pnpm snack doctor. So pnpm snack doctor. So let's run this. let's run this. Okay. So this Okay. So this did everything, we did everything, we did everything, we can see that node is up, Chrome is up, can see that node is up, Chrome is up, can see that node is up, Chrome is up, the session is active. Here's the session is active. Here's the session is active. Here's the evidence, uh, the last the evidence, uh, the last the evidence, uh, the last entry shows entry shows entry shows the bug, but the source is the bug, but the source is the bug, but the source is now, uh, this evidence is now, uh, this evidence is now, uh, this evidence is for code that for code that for code that no longer exists, so no longer exists, so no longer exists, so it couldn't be it couldn't be it couldn't be confirmed. That was a confirmed. That was a confirmed. That was a previous run.

previous run. previous run. Let's run it again Let's run it again Let's run it again . Same thing. . Same thing. . Same thing. Can anyone tell me Can anyone tell me Can anyone tell me what the bug is? Any ideas? what the bug is? Any ideas? Wrong email? Yeah, that's right Wrong email? Yeah, that's right . . . Wrong port. I think Wrong port. I think Wrong port. I think it was running on 62174. it was running on 62174. No, that was correct. Let me try Let me try again. We have less than a again. We have less than a again. We have less than a minute. For that, minute. For that, minute. For that, yes, you need to.

yes, you need to. yes, you need to. The tools make The tools make The tools make it optional, it optional, it optional, so you can go through so you can go through so you can go through all the exercises if you have all the exercises if you have all the exercises if you have it's not there and it'll it's not there and it'll it's not there and it'll work, but it's work, but it's work, but it's also a good opportunity also a good opportunity also a good opportunity to experiment with to experiment with to experiment with it if it is. it if it is. it if it is. Yeah. Yeah. Yeah. These are pretty cheap These are pretty cheap These are pretty cheap challenges. challenges. challenges. They give you $5 for They give you $5 for They give you $5 for free when you free when you free when you sign up, and I think sign up, and I think sign up, and I think that'll last you about that'll last you about that'll last you about 3 years. 3 years. 3 years. Yeah. Yeah. Okay. So, um, yeah, So, um, yeah, go ahead. go ahead. go ahead. Next. Yeah, this is Next. Yeah, this is Next. Yeah, this is Um, we ran this Um, we ran this Um, we ran this a few times. I think a few times. I think a few times. I think there was some problem with there was some problem with there was some problem with my output. I'll my output. I'll my output. I'll fix that. It's funny, fix that. It's funny, fix that. It's funny, I need I need I need verification tests for my verification tests for my verification tests for my fictional fictional fictional verification examples. But verification examples. But verification examples. But the results that the results that the results that can come up: can come up: can come up: called a bug, said it called a bug, said it called a bug, said it worked when it didn't, worked when it didn't, worked when it didn't, said it worked said it worked said it worked when it did, blamed a when it did, blamed a when it did, blamed a database that wasn't there, database that wasn't there, database that wasn't there, and there was no evidence of the and there was no evidence of the and there was no evidence of the correct bug. correct bug.

Yeah. It's just Yeah. It's just going back to the going back to the going back to the , uh, , uh, , uh, fictional example, fictional example, fictional example, of course, but your of course, but your of course, but your team may team may team may evaluate differently or evaluate differently or evaluate differently or have different criteria. have different criteria. have different criteria. It's extremely It's extremely It's extremely flexible. The most important thing flexible. The most important thing is to start and is to start and is to start and build something; build something; build something; create a baseline to create a baseline to create a baseline to evaluate, evaluate, evaluate, use it use it use it consistently, and consistently, and consistently, and develop it together as develop it together as develop it together as team, and then team, and then team, and then get to the point get to the point get to the point where your changes and PRs where your changes and PRs where your changes and PRs automatically automatically automatically check that check that check that score so you know score so you know score so you know if it's safe if it's safe if it's safe to implement the changes, to implement the changes, to implement the changes, if they make things better, if they make things better, if they make things better, if they if they if they make things worse, or if they don't make things worse, or if they don't change anything at all. change anything at all. Yeah. And every time Yeah. And every time Yeah. And every time it runs and it doesn't it runs and it doesn't it runs and it doesn't work, it's a new work, it's a new work, it's a new line that you can line that you can line that you can add to the map. It add to the map. It add to the map. It gives us an idea of ​​the gives us an idea of ​​the gives us an idea of ​​the steps we're taking, steps we're taking, steps we're taking, why we're doing it, and, you why we're doing it, and, you why we're doing it, and, you know, every failure know, every failure know, every failure gets us one step closer gets us one step closer gets us one step closer to the goal because we can to the goal because we can to the goal because we can fix it. So fix it. So fix it. So the first time it's just the first time it's just the first time it's just memory, meaning there's no memory, meaning there's no memory, meaning there's no data stored anywhere except memory data stored anywhere except memory data stored anywhere except memory . We . We . We enter the wrong enter the wrong enter the wrong email address email address email address . We lose . We lose . We lose our confirmation. our confirmation. our confirmation. Every time we Every time we Every time we can prove that we can prove that we can prove that we caught it, we caught it, we caught it, we don't encounter it again. don't encounter it again. But then there's the case But then there's the case of inflated of inflated of inflated expectations: a expectations: a expectations: a screenshot or a screenshot or a screenshot or a test run might tell test run might tell test run might tell us that everything went us that everything went us that everything went well when it didn't well when it didn't well when it didn't . Same thing with . Same thing with . Same thing with an error in the console. There an error in the console. There an error in the console. There might be nothing in the console might be nothing in the console might be nothing in the console , but it's in , but it's in , but it's in the receipt, or it's the receipt, or it's the receipt, or it's being intercepted by another being intercepted by another being intercepted by another before it before it before it hits the console.

hits the console. Yeah, something like that. Or Yeah, something like that. Or it could be nothing. it could be nothing. it could be nothing. There's nothing to look for. There's nothing to look for. There's nothing to look for. Everything works. And that's also a Everything works. And that's also a Everything works. And that's also a real answer. real answer. Yeah, if you Yeah, if you run run run diagnostics like we diagnostics like we diagnostics like we did, it did, it did, it showed previous showed previous showed previous examples. That's evidence examples. That's evidence examples. That's evidence of something that doesn't of something that doesn't of something that doesn't exist anymore. That's because your exist anymore. That's because your exist anymore. That's because your code could have code could have code could have changed significantly from changed significantly from changed significantly from yesterday to yesterday to yesterday to today. It could today. It could today. It could be other circumstances, be other circumstances, be other circumstances, like a known like a known like a known bug with a date. It could bug with a date. It could bug with a date. It could be a whole bunch of be a whole bunch of be a whole bunch of different things. So we different things. So we different things. So we want to build want to build want to build tools that tools that tools that are are are flexible enough not to flexible enough not to flexible enough not to get caught up in get caught up in get caught up in those kinds of issues. This all those kinds of issues. This all those kinds of issues. This all brings us back to the brings us back to the brings us back to the basic examples of basic examples of basic examples of unit unit unit testing, but it testing, but it testing, but it brings that kind of brings that kind of brings that kind of deterministic deterministic deterministic testing into the models testing into the models testing into the models so that they can so that they can so that they can self-heal and self-heal and self-heal and improve. improve. Absolutely. Okay, Absolutely. Okay, again, again, again, connecting this to what connecting this to what connecting this to what we did as we did as we did as human developers, human developers, human developers, right? What we right? What we right? What we saw earlier, how we saw earlier, how we saw earlier, how we tested the program, tested the program, tested the program, how we tested everything.

how we tested everything. how we tested everything. That's exactly what we were That's exactly what we were That's exactly what we were talking about, bringing talking about, bringing talking about, bringing this into code. So by this into code. So by this into code. So by bringing that into code and bringing that into code and bringing that into code and making it available in a making it available in a making it available in a deterministic deterministic CLI, we CLI, we can go through the can go through the can go through the same journey and have the same journey and have the same journey and have the same experience that same experience that same experience that we developers we developers we developers have when we're have when we're have when we're trying to trying to trying to fix something in Chrome fix something in Chrome fix something in Chrome late at night. And that's what late at night. And that's what late at night. And that's what opens up the possibilities opens up the possibilities opens up the possibilities for speed, for speed, for speed, development, and development, and development, and scaling the amount of scaling the amount of scaling the amount of code that you code that you code that you ship, while doing ship, while doing ship, while doing it in a more secure it in a more secure it in a more secure way. Again way. Again way. Again , you know, if , you know, if , you know, if you have a web app and you have a web app and you have a web app and you don't already have your own you don't already have your own you don't already have your own CLI verifier, this is CLI verifier, this is CLI verifier, this is one of the one of the one of the most effective most effective most effective tools to get tools to get tools to get started with started with started with your codebase, as well as a your codebase, as well as a your codebase, as well as a baseline baseline baseline assessment that assessment that assessment that demonstrates what a demonstrates what a demonstrates what a good good good performance of your performance of your performance of your app should look like. And also app should look like. And also app should look like. And also giving your giving your giving your agents the ability agents the ability agents the ability to see all aspects to see all aspects to see all aspects of performance, of performance, of performance, networking, traffic, networking, traffic, networking, traffic, rendering, etc. rendering, etc. rendering, etc. For example, giving them For example, giving them For example, giving them Chrome Chrome Chrome DevTools, if you're DevTools, if you're DevTools, if you're working in the web working in the web app space. If you're app space. If you're app space. If you're building native building native building native clients like clients like clients like iOS or OSX apps, there iOS or OSX apps, there iOS or OSX apps, there are are are alternative alternative alternative projects projects projects coming out every day coming out every day coming out every day that you can plug into them that you can plug into them that you can plug into them . But the . But the . But the common thread across common thread across common thread across them all is that you them all is that you them all is that you have to give have to give have to give your agent your agent your agent the ability to see the ability to see the ability to see absolutely everything that absolutely everything that absolutely everything that you normally see as a you normally see as a you normally see as a developer.

developer. developer. So, the more you So, the more you So, the more you give it, the give it, the give it, the more hands or eyes more hands or eyes you can you can you can give it, the more give it, the more give it, the more tasks it can tasks it can tasks it can take on for take on for take on for you. And then you you. And then you you. And then you just have to just have to just have to teach it to teach it to teach it to intuitively understand intuitively understand intuitively understand when and how to use all of that when and how to use all of that when and how to use all of that . . . Think about how Think about how Think about how you make you make you make decisions yourself: "Okay, decisions yourself: "Okay, decisions yourself: "Okay, this bug came in, how would this bug came in, how would this bug came in, how would I reproduce it?" I reproduce it?" I reproduce it?" Think about that. And Think about that. And Think about that. And if you can if you can if you can break it down into pieces break it down into pieces —and it depends on —and it depends on —and it depends on your application and the your application and the your application and the type of code you're type of code you're type of code you're writing—you might writing—you might writing—you might be able to make it be able to make it be able to make it so that it can " so that it can " think": "Oh, for us, think": "Oh, for us, think": "Oh, for us, this has a lot to do with this has a lot to do with this has a lot to do with logging." " logging." " Like, is the Like, is the cookie being set correctly?" cookie being set correctly?" "Is the expiration date being set "Is the expiration date being set "Is the expiration date being set , and all that , and all that , and all that ?" ?" ?" That's part of the job, That's part of the job, That's part of the job, right? Again, right? Again, right? Again, this applies to this applies to this applies to every industry and every every industry and every every industry and every niche that you're building for. niche that you're building for. software. But software. But the point is simply that the point is simply that the point is simply that the world where it was logical for us all to the world where it was logical for us all to write code by hand all the time is write code by hand all the time is at least at least at least evolving or evolving or evolving or at least at least at least changing rapidly. What changing rapidly. What changing rapidly. What fundamental fundamental fundamental systems principles systems principles systems principles can we apply can we apply can we apply now to speed up now to speed up now to speed up software development software development software development ? ? ? Yes. So yes. Working with Yes. So yes. Working with Yes. So yes. Working with voluminous voluminous voluminous call stacks. But the important thing is that call stacks. But the important thing is that call stacks. But the important thing is that you get you get you get the result, the result, the result, making sure that it making sure that it making sure that it provides detailed provides detailed provides detailed information about what information about what information about what happened, its happened, its happened, its state, so that all of that can be state, so that all of that can be state, so that all of that can be checked by checked by checked by the application or the model; the application or the model; the application or the model; and then you can and then you can and then you can run it again, run it again, run it again, fix it, run it fix it, run it fix it, run it again and make sure again and make sure that those changes really that those changes really that those changes really took effect and the took effect and the took effect and the problem is problem is problem is gone.

gone. And so, again, you And so, again, you can't can't can't overstate the value of overstate the value of overstate the value of keeping reports. keeping reports. keeping reports. So if you have So if you have So if you have launches, it could be the launches, it could be the launches, it could be the entire entire entire user journey. Let's say user journey. Let's say user journey. Let's say it's an e-commerce it's an e-commerce it's an e-commerce store. It's like, when store. It's like, when store. It's like, when they log in they log in they log in , they open , they open , they open the page, what happened, the page, what happened, the page, what happened, what was their journey? Did what was their journey? Did what was their journey? Did they see they see they see any errors at all any errors at all any errors at all ? Did ? Did ? Did they wait 7 seconds to they wait 7 seconds to they wait 7 seconds to see your catalog, see your catalog, see your catalog, well, while it well, while it well, while it loaded? Did loaded? Did loaded? Did they manage to add they manage to add they manage to add something to the cart? Was something to the cart? Was something to the cart? Was the cart abandoned? So the cart abandoned? So , keep all of that , keep all of that , keep all of that now, because each now, because each now, because each of those launches and of those launches and of those launches and reports can be reports can be reports can be fed into LLM, or you fed into LLM, or you fed into LLM, or you could even could even could even set up a system set up a system set up a system that runs all night long, that runs all night long, that runs all night long, looks for the last looks for the last looks for the last reports that were generated, reports that were generated, reports that were generated, aggregates them, and says, " aggregates them, and says, " aggregates them, and says, " We found We found We found eight cache module failures today eight cache module failures today eight cache module failures today ." So I'm ." So I'm ." So I'm going to, of course, going to, of course, submit a submit a merge request (PR) tonight to merge request (PR) tonight to merge request (PR) tonight to fix this fix this fix this properly, and properly, and properly, and then I have the then I have the then I have the tools I tools I tools I need to need to need to validate in validate in validate in that PR that the problem that PR that the problem that PR that the problem that I was trying to that I was trying to that I was trying to solve is fixed. solve is fixed. solve is fixed. And then there's the recommendations And then there's the recommendations And then there's the recommendations that we see from that we see from that we see from people who people who people who are reaching very are reaching very are reaching very high high high PR release scales without any bugs PR release scales without any bugs . They can also . They can also . They can also add things that add things that add things that are still under are still under are still under our human our human our human oversight. So not oversight. So not oversight. So not just screenshots of the just screenshots of the just screenshots of the feature, but a record of feature, but a record of feature, but a record of how I went through the how I went through the how I went through the app. That was app. That was app. That was my journey. That's how I my journey. That's how I my journey. That's how I felt. Those were the felt. Those were the felt. Those were the states, and I got there.

states, and I got there. states, and I got there. So if I So if I So if I wake up and wake up and wake up and try quickly try quickly try quickly review, you know, review, you know, review, you know, 15 important PRs from the 15 important PRs from the 15 important PRs from the morning, I can morning, I can morning, I can look at look at look at that first and make sure that that first and make sure that that first and make sure that everything is done and everything is done and everything is done and working. Then I working. Then I working. Then I can start looking at can start looking at can start looking at the code if that's something that I'm the code if that's something that I'm the code if that's something that I'm actually going to actually going to actually going to test as a human. test as a human. test as a human. And again, I And again, I And again, I think all of that is going to be think all of that is going to be think all of that is going to be incredibly valuable incredibly valuable incredibly valuable as we as we as we continue to continue to continue to see the see the see the evolution of evolution of self-improving tools. And you know, think about what And you know, think about what that means, because that means, because that means, because when we get to the when we get to the when we get to the point where point where self-improving tools self-improving tools become the standard and become the standard and become the standard and people start people start people start using them, there's going to be a using them, there's going to be a using them, there's going to be a significant significant significant advantage for those who advantage for those who advantage for those who have kept the clean have kept the clean have kept the clean data sets, saying, "This is data sets, saying, "This is data sets, saying, "This is what we're aiming for in what we're aiming for in what we're aiming for in this app, we this app, we this app, we want you to want you to want you to achieve this and achieve this and achieve this and improve it quickly improve it quickly improve it quickly ," right? If ," right? If ," right? If you don't have a mountain of you don't have a mountain of you don't have a mountain of data like that, and your data like that, and your data like that, and your competitors do, and competitors do, and competitors do, and they have similar apps, they have similar apps, they have similar apps, they can they can they can speed up their speed up their speed up their development cycle a lot. development cycle a lot. development cycle a lot. Yeah. And it just Yeah. And it just Yeah. And it just builds trust so that builds trust so that builds trust so that we can we can we can continue to continue to continue to improve our improve our improve our app, but app, but app, but we have we have we have the checks for everything the checks for everything the checks for everything that came before. So that came before. So that came before. So we have unit we have unit we have unit tests, evaluations, smoke tests, evaluations, smoke tests, evaluations, smoke tests, everything that tests, everything that tests, everything that our our our application needs, and we have a application needs, and we have a application needs, and we have a way for the model way for the model way for the model to interpret and to interpret and to interpret and run them, run them, run them, improving it so that improving it so that improving it so that when we when we when we add new add new add new features, we can features, we can features, we can run all the run all the run all the previous tests and previous tests and previous tests and make sure that we have make sure that we have make sure that we have n't broken anything, n't broken anything, n't broken anything, no regressions, and we no regressions, and we no regressions, and we can just can just can just climb this ladder.

climb this ladder. climb this ladder. a bunch of evidence, a bunch of evidence, a bunch of evidence, accumulating it as we accumulating it as we accumulating it as we go, and ensuring go, and ensuring go, and ensuring that our app works that our app works that our app works exactly as exactly as exactly as expected. And this expected. And this expected. And this actually becomes actually becomes actually becomes a specification that a specification that a specification that shows what shows what shows what our application does. Okay, our application does. Okay, our application does. Okay, cool. So, we have a cool. So, we have a cool. So, we have a little over 5 little over 5 little over 5 minutes. I'll minutes. I'll minutes. I'll pause and see if there pause and see if there pause and see if there are any more questions. Yes, are any more questions. Yes, are any more questions. Yes, sir. sir. sir. So, is the scale I would So, is the scale I would So, is the scale I would personally, um, I'm personally, um, I'm personally, um, I'm curious what you think about that curious what you think about that curious what you think about that . I would personally rather do . I would personally rather do . I would personally rather do away with away with away with skills, because in my skills, because in my skills, because in my experience, you know, it experience, you know, it experience, you know, it depends on the model depends on the model depends on the model whether it will whether it will whether it will actually follow actually follow actually follow that path and that path and that path and use them, use them, use them, as opposed to having as opposed to having as opposed to having CI actually CI actually CI actually force force force those commands to be executed. Yeah, that's exactly the those commands to be executed. Yeah, that's exactly the um, kind of answer or um, kind of answer or question in how question in how question in how you usually you usually you usually run these run these run these checks, right? checks, right? checks, right? Effectively. Yes, you Effectively. Yes, you Effectively. Yes, you can run them, can run them, can run them, you can run you can run you can run them regularly. You them regularly. You them regularly. You can make them can make them can make them part of CI, for example part of CI, for example . You can . You can . You can use use use various hooks, like the git various hooks, like the git hook, although that's probably the hook, although that's probably the hook, although that's probably the last thing I would last thing I would last thing I would resort to. We could resort to. We could resort to. We could do this with a cloud do this with a cloud do this with a cloud code hook. It code hook. It code hook. It depends on what depends on what depends on what your your your team uses and how they team uses and how they team uses and how they do it. I could do it. I could do it. I could create a full-fledged

create a full-fledged create a full-fledged extension to have everything extension to have everything extension to have everything run in the background run in the background run in the background and just and just and just serve up information as it serve up information as it serve up information as it goes. Um, there are goes. Um, there are goes. Um, there are a few different a few different a few different ways you can ways you can ways you can do this. But the do this. But the do this. But the easiest way easiest way easiest way to get everyone involved—and to get everyone involved—and to get everyone involved—and I can say this I can say this I can say this now—is to simply now—is to simply now—is to simply add an agents.md file, add an agents.md file, add an agents.md file, not a cloud.md file, to your not a cloud.md file, to your not a cloud.md file, to your application. And then application. And then application. And then write in it, for write in it, for write in it, for example: "when example: "when example: "when making changes, making changes, making changes, test everything test everything test everything by running this" or " by running this" or " by running this" or " every time you every time you every time you change something change something change something significant, or before significant, or before significant, or before opening a PR, do opening a PR, do opening a PR, do this." Well, whatever this." Well, whatever this." Well, whatever wording you wording you wording you want to want to want to put there. And it's an put there. And it's an put there. And it's an easy way to easy way to easy way to get everyone involved and they'll get everyone involved and they'll get everyone involved and they'll really have to really have to really have to make an effort to make an effort to make an effort to ignore it or ignore it or ignore it or avoid it. avoid it. avoid it. Yes, hooks are also Yes, hooks are also Yes, hooks are also useful if you useful if you useful if you are doing local are doing local are doing local development for the whole development for the whole development for the whole team, because you can team, because you can team, because you can share hooks with share hooks with share hooks with everyone. Yes, definitely. everyone. Yes, definitely. everyone. Yes, definitely. Yes. Yes. Yes. Well, CI is, I think, Well, CI is, I think, Well, CI is, I think, one of its features, one of its features, one of its features, but the tool itself but the tool itself is a rapid development is a rapid development is a rapid development that you get that you get that you get locally. So, I give locally. So, I give locally. So, I give my agent a my agent a my agent a task and I can task and I can task and I can just quickly just quickly just quickly run it, that is, run it, that is, run it, that is, let the agent let the agent let the agent work. You know, work. You know, work. You know, I just tell him what I I just tell him what I I just tell him what I want to get as a want to get as a want to get as a result. I want the result. I want the waitlist to work waitlist to work like this: I have the option like this: I have the option like this: I have the option to add an email to the to add an email to the to add an email to the waitlist, after waitlist, after waitlist, after which it is which it is which it is saved and I can saved and I can saved and I can check it. So, check it. So, check it. So, here's how you here's how you here's how you check it. This check it. This check it. This goes into such and such a goes into such and such a goes into such and such a table, and so on.

table, and so on. table, and so on. Well, whatever it is. Um, Well, whatever it is. Um, Well, whatever it is. Um, yes. No, good yes. No, good yes. No, good question. Of course. question. Of course. Yes. So, what is Yes. So, what is the advantage of running in CI the advantage of running in CI the advantage of running in CI compared to compared to compared to running locally? running locally? Of course. The benefits Of course. The benefits are definitely a are definitely a are definitely a faster faster faster feedback loop, feedback loop, feedback loop, which allows us which allows us which allows us to work faster and to work faster and to work faster and allows me allows me allows me to parallelize to parallelize to parallelize tasks since I'm tasks since I'm tasks since I'm no longer working no longer working no longer working on just one thing. I'm on just one thing. I'm on just one thing. I'm working on at least working on at least working on at least now I have now I have now I have three agents running. It's very three agents running. It's very three agents running. It's very small for me, but I'm small for me, but I'm small for me, but I'm at a conference. at a conference. at a conference. Usually there are 10 to 15 of them Usually there are 10 to 15 of them . . . He is really He is really He is really resting today. resting today. resting today. Um, and the whole idea is that Um, and the whole idea is that Um, and the whole idea is that I'm launching it and I I'm launching it and I I'm launching it and I don't want it to be waiting don't want it to be waiting don't want it to be waiting for me, saying, "Yes, for me, saying, "Yes, for me, saying, "Yes, I'm done." And then I I'm done." And then I I'm done." And then I check, and he says, check, and he says, "No, that's actually "No, that's actually "No, that's actually wrong." and then wrong." and then wrong." and then I have to I have to I have to start it again, start it again, start it again, I get distracted by I get distracted by I get distracted by another agent, and so I another agent, and so I another agent, and so I continue this cycle. I'm continue this cycle. I'm continue this cycle. I'm just constantly just constantly just constantly switching between switching between switching between projects. So I want to projects. So I want to projects. So I want to give him the give him the give him the tools so he tools so he tools so he can prove to himself that he's can prove to himself that he's can prove to himself that he's done it, and then done it, and then done it, and then prove it to me—that is, prove it to me—that is, prove it to me—that is, show me evidence that show me evidence that show me evidence that I can trust. So that I can trust. So that I can trust. So that when I come back to when I come back to when I come back to it—in 5 minutes, an it—in 5 minutes, an it—in 5 minutes, an hour, or a few hour, or a few hour, or a few hours—I know it did its hours—I know it did its best to get best to get to the end, and I don't to the end, and I don't to the end, and I don't have to have to have to give it hints again and

give it hints again and give it hints again and start all over again start all over again start all over again , because that's... that's not the , because that's... that's not the , because that's... that's not the most fun part of most fun part of most fun part of my job. my job. my job. Yes, I would say that one Yes, I would say that one Yes, I would say that one could put it could put it could put it another way: while we don't another way: while we don't another way: while we don't manage to achieve manage to achieve manage to achieve this perfectly every day, this perfectly every day, this perfectly every day, I want all the I want all the areas I started that day that I areas I started that day that I felt were important for the felt were important for the felt were important for the release to be release to be release to be completed. I want to completed. I want to completed. I want to go back to them go back to them go back to them at the end of the day and at the end of the day and at the end of the day and see that they are, in see that they are, in see that they are, in essence, correctly essence, correctly essence, correctly executed. I definitely don't executed. I definitely don't executed. I definitely don't want to work all want to work all want to work all day long while day long while day long while they're working in the background, they're working in the background, they're working in the background, doing other doing other doing other things, going to things, going to things, going to meetings, and then meetings, and then meetings, and then coming back and coming back and coming back and making each of making each of making each of them move step by them move step by them move step by step, because that step, because that step, because that slows down the work considerably and slows down the work considerably and slows down the work considerably and prevents me from prevents me from prevents me from moving faster. In moving faster. In moving faster. In other words, I other words, I other words, I rely on CI and the rely on CI and the rely on CI and the tests running tests running tests running in CI, as well as in CI, as well as in CI, as well as code review. code review. code review. How many people here How many people here How many people here use use use automated automated automated checks like Reptile, checks like Reptile, checks like Reptile, Code Rabbit, Codo or similar? Code Rabbit, Codo or similar? Code Rabbit, Codo or similar? Yes. I rely on Yes. I rely on Yes. I rely on them. We them. We them. We use Reptile use Reptile use Reptile and it gives me an and it gives me an and it gives me an X out of five. If it X out of five. If it X out of five. If it scores five out of five scores five out of five scores five out of five and all my CI tests and all my CI tests and all my CI tests pass, I feel pass, I feel pass, I feel completely confident that I completely confident that I completely confident that I won't break anything in won't break anything in won't break anything in production after production after production after release. We have reached release. We have reached release. We have reached this level of trust this level of trust this level of trust in Reptile. The most important thing in Reptile. The most important thing in Reptile. The most important thing for me is for me is for me is to reach this state faster and to reach this state faster and with fewer with fewer steps.

steps. steps. With fewer With fewer With fewer steps, yes. But steps, yes. But steps, yes. But you know, I can add a you know, I can add a you know, I can add a whole, say, 10,000 whole, say, 10,000 whole, say, 10,000 lines of absolutely lines of absolutely lines of absolutely nothing, and it does nothing, and it does nothing, and it does n't do anything, and great— n't do anything, and great— it passes the test. it passes the test. it passes the test. Reptile got a five out of Reptile got a five out of Reptile got a five out of five for some five for some five for some terrible reason, terrible reason, terrible reason, it goes into it goes into it goes into production and production and production and does nothing; it doesn't does nothing; it doesn't does nothing; it doesn't break anything, but now break anything, but now break anything, but now we have an extra 10,000 we have an extra 10,000 we have an extra 10,000 lines of nothing. So, lines of nothing. So, lines of nothing. So, it also helps it also helps it also helps me make sure that me make sure that me make sure that I'm telling the agent exactly what I I'm telling the agent exactly what I I'm telling the agent exactly what I want and it achieves want and it achieves want and it achieves the result without any the result without any the result without any unnecessary extra unnecessary extra unnecessary extra code; he code; he code; he stays on stays on stays on course and also knows course and also knows course and also knows how to correct himself how to correct himself how to correct himself to stay on to stay on to stay on task while task while task while I'm busy with other I'm busy with other I'm busy with other things. things. Okay, maybe we Okay, maybe we have time for one more have time for one more have time for one more quick question, but quick question, but quick question, but I think we...Yes, I think we...Yes, I think we...Yes, yes, sir. yes, sir. Yes. Yes. Aha. You Yes. Yes. Aha. You can also, depending on can also, depending on can also, depending on your stack, your stack, your stack, use use use the services of different the services of different the services of different providers to providers to providers to easily easily easily jump to a jump to a preview first and then preview first and then test everything in another test everything in another test everything in another way. So then way. So then way. So then we'll move on to we'll move on to we'll move on to production and production and production and start monitoring right away.

start monitoring right away. start monitoring right away. This is another pattern This is another pattern This is another pattern we're starting to we're starting to we're starting to see: when people see: when people see: when people have a deployment that's have a deployment that's have a deployment that's implemented implemented implemented primarily through primarily through primarily through agents, there's agents, there's agents, there's at least one agent at least one agent at least one agent specifically specifically specifically responsible for responsible for responsible for that release into production that release into production , which tracks all the , which tracks all the , which tracks all the metrics, DataDog, and metrics, DataDog, and metrics, DataDog, and feature switches, feature switches, feature switches, you know? Being you know? Being you know? Being able to say that this is able to say that this is able to say that this is actually bad, and actually bad, and actually bad, and we roll it back. So we roll it back. So we roll it back. So I really understand and I really understand and I really understand and feel the anxiety feel the anxiety feel the anxiety that we all have that we all have that we all have when we when we when we face this, but I face this, but I face this, but I see a way through it see a way through it see a way through it because it seems because it seems because it seems inevitable. Yes. Aha. inevitable. Yes. Aha. inevitable. Yes. Aha. That's all the time That's all the time That's all the time we had, but if we had, but if we had, but if you have any more questions, we'd you have any more questions, we'd you have any more questions, we'd be happy to be happy to be happy to chat. We chat. We chat. We will be in the expo hall will be in the expo hall will be in the expo hall at booth number six at booth number six , on the right as you , on the right as you , on the right as you enter. Come enter. Come enter. Come to us. We would be happy to to us. We would be happy to to us. We would be happy to chat, chat, chat, show you what we show you what we show you what we are working on, and are working on, and are working on, and talk to you about talk to you about talk to you about anything that interests you. anything that interests you. anything that interests you. Yes. Thank you very much for Yes. Thank you very much for Yes. Thank you very much for coming, for your coming, for your coming, for your attention and everything else. attention and everything else. attention and everything else. Thank you very much.