Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI Lab

Can everyone hear me? Can everyone hear me? Okay, great. So, first Okay, great. So, first Okay, great. So, first of all, thank you to everyone who of all, thank you to everyone who of all, thank you to everyone who joined. My joined. My joined. My name is Felipe, and I'm a name is Felipe, and I'm a name is Felipe, and I'm a technical specialist at the technical specialist at the technical specialist at the Amazon AGI Lab. As part of Amazon AGI Lab. As part of Amazon AGI Lab. As part of my role, I my role, I my role, I help us help us help us create the next create the next create the next generation. I manage the generation. I manage the next generation product development programs that our that our that our lab is developing. And lab is developing. And lab is developing. And today I will tell today I will tell today I will tell you about developing you about developing you about developing evaluations (Evals) that evaluations (Evals) that evaluations (Evals) that gain gain gain customer trust. Before we customer trust. Before we customer trust. Before we talk about the grades, talk about the grades, talk about the grades, I need to give I need to give I need to give you some context you some context you some context about what our about what our about what our lab has lab has lab has been up to this past been up to this past been up to this past year. So, it all year. So, it all year. So, it all started in March of started in March of started in March of last year when last year when last year when we launched the we launched the we launched the research version of research version of research version of Nova Act. For context, Nova Act. For context, Nova Act. For context, Nova Act is a tool Nova Act is a tool Nova Act is a tool we created in we created in we created in our lab. This is a our lab. This is a our lab. This is a tool that tool that tool that helps you helps you helps you create browser create browser create browser agents. So, agents. So, agents. So, basically, anything you basically, anything you basically, anything you do in a web browser, do in a web browser, do in a web browser, you can you can you can automate automate automate with Nova Act. And with Nova Act. And with Nova Act. And within this within this within this research version, research version, research version, our main goal our main goal our main goal was to release the was to release the was to release the product to get as product to get as product to get as much much much feedback from our feedback from our feedback from our customers as possible.

customers as possible. customers as possible. We launched it in March. We launched it in March. Then in July, we Then in July, we made an additional made an additional made an additional improvement to our improvement to our improvement to our product, which was product, which was product, which was to develop a completely to develop a completely to develop a completely new new new developer experience. This was developer experience. This was developer experience. This was because during the because during the because during the first few first few first few months we noticed months we noticed months we noticed some gaps in the some gaps in the some gaps in the developer experience developer experience developer experience and we wanted to and we wanted to and we wanted to make sure make sure make sure customers had the customers had the customers had the best possible best possible best possible agent building experience. agent building experience. Then in December we Then in December we finally launched finally launched finally launched our service on AWS. So our service on AWS. So our service on AWS. So today, Nova Act is a today, Nova Act is a today, Nova Act is a first-class service first-class service first-class service on AWS that anyone on AWS that anyone on AWS that anyone can use can use can use to build to build to build industrial- industrial- industrial- grade agents. And during grade agents. And during grade agents. And during this period, I was this period, I was this period, I was actually responsible for the actually responsible for the actually responsible for the direction of direction of direction of client development. What is client development. What is client development. What is customer development, customer development, customer development, just for context? just for context? just for context? This means that I This means that I This means that I worked with all of worked with all of worked with all of our customers every day to our customers every day to our customers every day to understand what understand what understand what their problem was and their problem was and their problem was and how we help them how we help them how we help them solve it. This has been solve it. This has been solve it. This has been my main role my main role my main role for the past for the past for the past year. Now year. Now year. Now let's move on to the let's move on to the let's move on to the problem we will problem we will problem we will discuss today.

discuss today. I like to call this the I like to call this the benchmark illusion. benchmark illusion. So, all of you who So, all of you who work on AI are work on AI are work on AI are probably working and probably working and probably working and building things building things building things based on benchmarks, based on benchmarks, based on benchmarks, right? These can be right? These can be right? These can be public benchmarks public benchmarks public benchmarks that that that everyone uses, and everyone everyone uses, and everyone everyone uses, and everyone tries to tries to tries to optimize their optimize their optimize their products for them. And products for them. And products for them. And also benchmarks that also benchmarks that also benchmarks that you create yourself, you create yourself, you create yourself, your own synthetic your own synthetic your own synthetic data with which data with which data with which you evaluate your you evaluate your you evaluate your product. And the main product. And the main product. And the main goal here is goal here is to optimize, right? to optimize, right? to optimize, right? You want to be You want to be You want to be as good as possible in these as good as possible in these as good as possible in these benchmarks. But then benchmarks. But then , when you release , when you release , when you release it into production and it into production and it into production and your customers your customers your customers start start start using it, this is what using it, this is what using it, this is what happens, happens, happens, right? They start to right? They start to right? They start to see problems, and see problems, and see problems, and basically everything you did basically everything you did basically everything you did n't expect them to do, n't expect them to do, n't expect them to do, they try they try they try to do, and they break to do, and they break to do, and they break your product. And this your product. And this your product. And this happens because happens because happens because most of us most of us most of us work with estimates work with estimates work with estimates that are static. that are static. So, what are So, what are static estimates?

static estimates? static estimates? Let's talk a little bit Let's talk a little bit Let's talk a little bit about about about what the standard product what the standard product what the standard product development process is development process is development process is , shall we? , shall we? The first thing you do is get requirements is get requirements is get requirements from your from your from your product team product team product team and start and start and start development, right? development, right? development, right? I hope I hope I hope everyone does that. Then, everyone does that. Then, everyone does that. Then, when the product is in when the product is in when the product is in good condition, good condition, good condition, the next thing you the next thing you the next thing you do is do is do is run your run your run your evaluations. Again, evaluations. Again, evaluations. Again, public benchmarks, public benchmarks, public benchmarks, synthetic data that synthetic data that synthetic data that you generated, and you you generated, and you you generated, and you use it. use it. use it. When you start to When you start to When you start to feel feel feel confident, you confident, you confident, you say, "Okay, now I say, "Okay, now I say, "Okay, now I can let it out." And can let it out." And can let it out." And you release it to you release it to you release it to customers. But if customers. But if customers. But if you have a static score, you have a static score, you have a static score, what is the only thing you can what is the only thing you can what is the only thing you can do after that? do after that? In essence, it's just hope . Yes? You just . Yes? You just . Yes? You just hope that your hope that your hope that your ratings accurately ratings accurately ratings accurately reflect what reflect what reflect what customers will do customers will do customers will do with the product, and if they with the product, and if they with the product, and if they don't, then everything starts to don't, then everything starts to don't, then everything starts to fall apart, fall apart, fall apart, right? And that is precisely where the right? And that is precisely where the right? And that is precisely where the gap lies. gap lies. gap lies. This is exactly what we will This is exactly what we will This is exactly what we will discuss today discuss today discuss today and what we have been doing in and what we have been doing in and what we have been doing in our lab our lab our lab to solve this to solve this to solve this problem. And a little problem. And a little problem. And a little spoiler: this is not about spoiler: this is not about spoiler: this is not about the best benchmarks. It's the best benchmarks. It's the best benchmarks. It's about closing the about closing the about closing the feedback loop with feedback loop with feedback loop with the customer. So what we the customer. So what we the customer. So what we want you to want you to want you to do here is do here is do here is get feedback from get feedback from get feedback from the customer and close the the customer and close the the customer and close the loop. So, with loop. So, with loop. So, with this in mind, in our this in mind, in our this in mind, in our lab we lab we lab we proposed an proposed an proposed an evaluation flywheel ( evaluation flywheel ( eval flywheel). If you're not eval flywheel). If you're not eval flywheel). If you're not familiar with familiar with familiar with the terminology, a the terminology, a the terminology, a flywheel is essentially a flywheel is essentially a flywheel is essentially a cycle that you cycle that you cycle that you keep repeating.

keep repeating. And the first step of these And the first step of these assessments is defining assessments is defining assessments is defining success. And here's the success. And here's the success. And here's the important thing: the definition of important thing: the definition of important thing: the definition of success is not what success is not what success is not what you consider success. you consider success. you consider success. This is what your client This is what your client This is what your client considers success, considers success, considers success, okay? So, the okay? So, the okay? So, the next step: next step: next step: once you understand once you understand once you understand what what what the customer really needs and the customer really needs and the customer really needs and what problem you what problem you what problem you are trying to solve are trying to solve , you move on to , you move on to , you move on to collecting signals. And we, collecting signals. And we, collecting signals. And we, as engineers, what as engineers, what as engineers, what comes to comes to comes to mind first? This is mind first? This is mind first? This is a toolkit, right? a toolkit, right? a toolkit, right? Let's collect Let's collect Let's collect metrics, get metrics, get metrics, get as much data as possible as much data as possible as much data as possible about what your client is doing about what your client is doing about what your client is doing . But one very . But one very . But one very important part that important part that important part that we miss is that we miss is that we miss is that you have to you have to you have to talk to your talk to your talk to your customer. Yes, customer. Yes, customer. Yes, communicate with your communicate with your communicate with your client. Go, client. Go, client. Go, make appointments make appointments , attend them, and , attend them, and , attend them, and understand what exactly understand what exactly understand what exactly they are trying they are trying they are trying to do. Your data to do. Your data to do. Your data may be a little may be a little may be a little misleading. Maybe misleading. Maybe misleading. Maybe you don't understand exactly what you don't understand exactly what you don't understand exactly what problem problem your client is trying to solve. The third your client is trying to solve. The third step is step is step is gap diagnosis. It's simple.

gap diagnosis. It's simple. gap diagnosis. It's simple. Essentially, you collect Essentially, you collect Essentially, you collect all the signals from all the signals from all the signals from customers. This can customers. This can customers. This can be a toolkit, be a toolkit, be a toolkit, as well as your own as well as your own as well as your own observations and observations and observations and insights from meetings insights from meetings insights from meetings with them. And you with them. And you with them. And you start start start to classify them. to classify them. to classify them. When I say " When I say " classify," there classify," there classify," there are generally three are generally three are generally three categories that can be categories that can be categories that can be distinguished from these distinguished from these distinguished from these discussions. The first is discussions. The first is , in essence, feedback about , in essence, feedback about , in essence, feedback about your model. Where exactly is your model. Where exactly is your model. Where exactly is your model failing? your model failing? your model failing? And that promotes development, And that promotes development, And that promotes development, right? Your model. right? Your model. right? Your model. The second point is a The second point is a The second point is a task for your task for your task for your engineering team. engineering team. That is, essentially, where is That is, essentially, where is the gap in the system the gap in the system the gap in the system that needs to be fixed. that needs to be fixed. that needs to be fixed. This is a more traditional This is a more traditional This is a more traditional error correction. error correction. error correction. And the third point And the third point And the third point concerns the concerns the concerns the product itself. What are you product itself. What are you product itself. What are you missing from a missing from a missing from a product perspective? For example, product perspective? For example, product perspective? For example, does your customer understand does your customer understand what your product is supposed to what your product is supposed to what your product is supposed to solve? Is solve? Is your your product positioned correctly? This is very product positioned correctly? This is very product positioned correctly? This is very important. So, once important. So, once important. So, once you you you have classified have classified have classified the gaps, you the gaps, you the gaps, you move on to move on to move on to decision making.

decision making. Decision making is the Decision making is the prioritization of prioritization of prioritization of tasks for the science, tasks for the science, tasks for the science, engineering, and engineering, and engineering, and product teams. product teams. product teams. It's simple, It's simple, It's simple, right? After that, right? After that, right? After that, you just you just you just keep keep keep repeating this cycle repeating this cycle repeating this cycle over and over again. With over and over again. With over and over again. With that in mind, that in mind, that in mind, we we we have repeated this cycle have repeated this cycle have repeated this cycle many times in our lab, many times in our lab, many times in our lab, starting in March of starting in March of starting in March of last year. I last year. I last year. I also noticed that also noticed that also noticed that while working on a while working on a while working on a product, each product, each product, each client develops their client develops their client develops their own way of own way of own way of using it. What does this using it. What does this using it. What does this mean? My customer mean? My customer mean? My customer is growing is growing is growing by using my by using my by using my product, right? product, right? product, right? So, there are four things you So, there are four things you So, there are four things you can learn can learn can learn while a customer while a customer while a customer is using your is using your is using your product. The first is what product. The first is what product. The first is what you do during your you do during your you do during your first meeting with a first meeting with a first meeting with a client. This is a search for client. This is a search for use cases. Here you use cases. Here you want to understand want to understand want to understand what their what their what their problem is. So what is the problem is. So what is the problem is. So what is the customer's problem, and are customer's problem, and are customer's problem, and are you confident that you confident that you confident that your solution your solution your solution fixes it? Then fixes it? Then fixes it? Then you will have your you will have your you will have your first few users.

first few users. You You want to get the want to get the want to get the main signal from these early adopters—which is main signal from these early adopters—which is main signal from these early adopters—which is even possible. This is a even possible. This is a even possible. This is a time when customers time when customers time when customers will try will try will try crazy things. You will crazy things. You will crazy things. You will find situations where find situations where find situations where you can't do anything you can't do anything you can't do anything to help them, to help them, to help them, right? But you will also right? But you will also right? But you will also find problems find problems find problems that customers that customers that customers solve solve solve with your with your with your product, and often you product, and often you product, and often you will be surprised because you would will be surprised because you would will be surprised because you would never have never have never have imagined it. The third point imagined it. The third point is when you is when you is when you start start start scaling, right? scaling, right? scaling, right? You get your first You get your first You get your first users, you users, you users, you start start start to scale, you to scale, you to scale, you attract more attract more attract more customers, and you can customers, and you can customers, and you can see some see some see some patterns, right? patterns, right? patterns, right? What are these What are these What are these patterns? This is patterns? This is patterns? This is essentially your primary essentially your primary essentially your primary use case for use case for use case for the product. This is the the product. This is the the product. This is the case, the problem case, the problem case, the problem that everyone is trying that everyone is trying that everyone is trying to solve with to solve with to solve with your your your product. And lastly, product. And lastly, product. And lastly, this is a look into the future this is a look into the future —your gaps in the —your gaps in the —your gaps in the product. While working with the product. While working with the product. While working with the first first first users, as well users, as well users, as well as after you start as after you start as after you start scaling, you will scaling, you will scaling, you will also discover some also discover some also discover some flaws in your flaws in your flaws in your product. Your product. Your product. Your development team development team development team will help will help will help prioritize them, and this prioritize them, and this prioritize them, and this will become your foundation will become your foundation will become your foundation for the future. What should be for the future. What should be for the future. What should be created next? As I created next? As I created next? As I said, I have worked said, I have worked said, I have worked with many with many with many clients along this clients along this clients along this path, and I want path, and I want path, and I want to share a few to share a few to share a few lessons: some of lessons: some of lessons: some of them are obvious, but them are obvious, but them are obvious, but others are very insightful.

others are very insightful. The first is what I The first is what I call the “ call the “ call the “ trust gap.” So, what is the trust gap.” So, what is the trust gap.” So, what is the “trust gap”? “trust gap”? Imagine that you Imagine that you have created a product, have created a product, have created a product, are ready to launch, are ready to launch, are ready to launch, have measured its have measured its have measured its reliability and reliability and reliability and have received a score of 80%. have received a score of 80%. It seems pretty good, It seems pretty good, right? Like, okay, I right? Like, okay, I right? Like, okay, I can give this to can give this to can give this to the client and they'll the client and they'll the client and they'll start start start using it, right? using it, right? using it, right? But when you tell But when you tell But when you tell a customer that a customer that a customer that the reliability is 80% the reliability is 80% , they think, "I can't , they think, "I can't , they think, "I can't trust this trust this trust this product." I can't product." I can't product." I can't trust. Why? Because trust. Why? Because trust. Why? Because at 80% reliability, at 80% reliability, at 80% reliability, the client thinks, "Okay, the client thinks, "Okay, the client thinks, "Okay, if it's 80%, I still if it's 80%, I still if it's 80%, I still have to follow up have to follow up have to follow up with this agent." If with this agent." If with this agent." If it fails, I'll it fails, I'll it fails, I'll have to have to have to do the work do the work do the work manually. So, essentially, manually. So, essentially, manually. So, essentially, it's more work for me it's more work for me it's more work for me , not less.

, not less. , not less. I need to I need to I need to configure this configure this configure this agent, monitor it agent, monitor it agent, monitor it , and also do some , and also do some , and also do some manual work. But when you manual work. But when you manual work. But when you reach about 90% reach about 90% reliability, the reliability, the reliability, the customer's perception customer's perception customer's perception changes: he thinks changes: he thinks that 90% is already reliable that 90% is already reliable , and you can trust , and you can trust , and you can trust the product to perform the product to perform the product to perform tasks and completely tasks and completely tasks and completely transfer the transfer the transfer the workflow. So, workflow. So, workflow. So, reliability is a reliability is a reliability is a zero-or- zero-or- zero-or- one principle. Either you one principle. Either you one principle. Either you trust or you don't. There is trust or you don't. There is trust or you don't. There is nothing nothing nothing in between here. The second in between here. The second in between here. The second conclusion is “false conclusion is “false conclusion is “false transparency.” So, transparency.” So, transparency.” So, every time you every time you every time you start working start working start working with a client, you with a client, you with a client, you need to be need to be need to be upfront about upfront about upfront about three things that three things that three things that your product does. The your product does. The first thing is easy. first thing is easy. Everyone loves to do this Everyone loves to do this . So, what works? You . So, what works? You . So, what works? You are actually showing are actually showing are actually showing the client: I am really the client: I am really the client: I am really good at this. This is the good at this. This is the good at this. This is the level of reliability. You level of reliability. You level of reliability. You can trust can trust can trust my product.

my product. my product. Just Just Just use my use my use my agent and he agent and he agent and he will provide you with the will provide you with the will provide you with the reliability you need reliability you need . Second point: here's . Second point: here's . Second point: here's what I'm doing to what I'm doing to what I'm doing to improve my improve my improve my product. Okay, there are product. Okay, there are product. Okay, there are some gaps that some gaps that some gaps that I'm aware of, but I'm aware of, but I'm aware of, but I'm currently investing time to I'm currently investing time to I'm currently investing time to make them better. And the make them better. And the make them better. And the third thing is the third thing is the third thing is the most difficult. Nobody most difficult. Nobody most difficult. Nobody wants to do that. wants to do that. wants to do that. This is something that goes beyond This is something that goes beyond This is something that goes beyond the limits of possibility. So the limits of possibility. So the limits of possibility. So you really need you really need you really need to be transparent. If to be transparent. If to be transparent. If something doesn't work in something doesn't work in something doesn't work in your product, your product, your product, just go and just go and just go and tell the tell the tell the customer. Don't customer. Don't customer. Don't let them let them let them try. Otherwise, try. Otherwise, try. Otherwise, because there are because there are because there are so many so many so many tools on the market now, if you tools on the market now, if you tools on the market now, if you show them something that doesn't show them something that doesn't show them something that doesn't work, they'll just work, they'll just work, they'll just forget about our forget about our forget about our product and move on product and move on product and move on to the next one. So to the next one. So to the next one. So be transparent. And the be transparent. And the be transparent. And the main lesson here: main lesson here: main lesson here: transparency about transparency about transparency about product limitations product limitations product limitations builds trust builds trust builds trust more than higher more than higher more than higher performance. This is a very performance. This is a very performance. This is a very interesting discovery, is interesting discovery, is interesting discovery, is n't it? Be n't it? Be n't it? Be upfront about what upfront about what upfront about what your product can do your product can do your product can do and, more and, more and, more importantly, what it importantly, what it importantly, what it can't do. This is how you can't do. This is how you can't do. This is how you gain gain gain customer trust. Lesson Three customer trust. Lesson Three : If you work : If you work : If you work very closely with very closely with very closely with customers, you can customers, you can customers, you can find many find many use cases that you would use cases that you would never expect never expect . Here are some . Here are some . Here are some examples: Amazon Leo is a examples: Amazon Leo is a examples: Amazon Leo is a satellite internet satellite internet company from Amazon.

company from Amazon. Working with them, we Working with them, we gained important gained important gained important insights into creating insights into creating insights into creating functions for functions for functions for caching trajectories caching trajectories . So, instead of . So, instead of performing inference every time performing inference every time they they they use our use our use our product, we cache the product, we cache the product, we cache the trajectories. If that trajectories. If that trajectories. If that fails, we fails, we fails, we simply revert simply revert simply revert to the model. This to the model. This to the model. This way we optimize way we optimize way we optimize costs for them. costs for them. costs for them. The second lesson is with Hertz, a The second lesson is with Hertz, a car rental company. They car rental company. They also also also used our used our used our product for product for product for QA automation. And we QA automation. And we QA automation. And we learned that their learned that their learned that their company had QA engineers company had QA engineers who were very who were very who were very technical and technical and technical and capable of creating capable of creating capable of creating scripts in Python and scripts in Python and scripts in Python and other other other programming languages. However, programming languages. However, programming languages. However, some people on the some people on the some people on the QA team are not so tech- QA team are not so tech- QA team are not so tech- savvy. So it was that savvy. So it was that savvy. So it was that same feedback same feedback same feedback that inspired us that inspired us that inspired us to create a completely to create a completely to create a completely non-technical experience non-technical experience non-technical experience with the product, with the product, with the product, which would allow which would allow which would allow more more more customers customers customers to use it to use it . The latest is a startup . The latest is a startup . The latest is a startup called Solute. Solute called Solute. Solute is essentially an is essentially an is essentially an RPA company. If RPA company. If RPA company. If you're not familiar with the you're not familiar with the you're not familiar with the term, RPA is the term, RPA is the term, RPA is the automation of automation of automation of repetitive repetitive repetitive processes. They processes. They processes. They offer offer offer tools to tools to tools to automate the automate the automate the processes of other processes of other processes of other companies. So imagine the companies. So imagine the companies. So imagine the variety of variety of uses they uses they could have. That's could have. That's could have. That's a lot of different a lot of different usage scenarios. And we usage scenarios. And we realized that we could realized that we could realized that we could provide more provide more provide more flexibility to our flexibility to our flexibility to our product for their product for their product for their customizations, customizations, customizations, especially in the especially in the especially in the execution stack of our execution stack of our execution stack of our tool. The lessons

tool. The lessons tool. The lessons here are less important than here are less important than here are less important than understanding that you understanding that you understanding that you will learn a lot will learn a lot will learn a lot about what exactly about what exactly your client is trying to do, right? And what are the next And what are the next features you should features you should features you should implement in your implement in your implement in your product? And so, product? And so, product? And so, after running this flywheel after running this flywheel after running this flywheel several times, we several times, we several times, we derived four derived four derived four principles for it. principles for it. First: create First: create assessment scenarios assessment scenarios assessment scenarios based on real based on real based on real work, not imagination. work, not imagination. Again, Again, generating generating generating synthetic data synthetic data synthetic data can be very can be very can be very important early in the important early in the important early in the process, but once process, but once customer signals emerge, customer signals emerge, use them use them use them for evaluation. for evaluation. Second: classify Second: classify everything into areas that everything into areas that everything into areas that need action. need action. need action. Again, product issues Again, product issues , engineering or , engineering or , engineering or research— research— make sure you make sure you prioritize them correctly for your prioritize them correctly for your team. Third: team. Third: team. Third: share share share opportunities honestly. Again opportunities honestly. Again opportunities honestly. Again , transparency , transparency , transparency builds trust. builds trust. builds trust. Be sure to Be sure to Be sure to let the customer know let the customer know , especially about what , especially about what , especially about what your product does your product does your product does n't do n't do n't do well.

well. Third (fourth) Third (fourth) point: start the point: start the point: start the flywheel as flywheel as flywheel as quickly as possible. It's quickly as possible. It's quickly as possible. It's simple: the faster you simple: the faster you simple: the faster you get this get this get this flywheel going, the faster flywheel going, the faster flywheel going, the faster your your your product improves, right? It product improves, right? It product improves, right? It seems obvious, right? seems obvious, right? seems obvious, right? But remember But remember But remember this: get your this: get your this: get your flywheel going as flywheel going as flywheel going as quickly as possible. And with these quickly as possible. And with these quickly as possible. And with these four principles, four principles, four principles, you will see your you will see your you will see your grades become grades become grades become smarter every week. smarter every week. But it's very important: But it's very important: if you don't see if you don't see if you don't see your estimates getting your estimates getting your estimates getting smarter every week, smarter every week, smarter every week, it means they're getting it means they're getting it means they're getting stale. stale. stale. Of course? And to Of course? And to Of course? And to conclude my speech, I conclude my speech, I conclude my speech, I want you to take want you to take want you to take two thoughts with you. two thoughts with you. two thoughts with you. Only two. First: Only two. First: Only two. First: the assessment should the assessment should the assessment should reflect what reflect what reflect what the client cares about, not what the client cares about, not what the client cares about, not what you think you think you think they care about. Good? It they care about. Good? It they care about. Good? It all comes down to all comes down to all comes down to getting getting getting customer feedback so that your customer feedback so that your customer feedback so that your ratings reflect ratings reflect ratings reflect their real their real their real issues. Second: issues. Second: issues. Second: ratings are not ratings are not ratings are not created once created once created once and for all. It's a and for all. It's a and for all. It's a flywheel that flywheel that flywheel that evolves with evolves with evolves with your customer. One your customer. One your customer. One note: note: note: remember the remember the remember the customer journey? As customer journey? As customer journey? As a customer moves through a customer moves through a customer moves through this journey, their goals for this journey, their goals for this journey, their goals for using your using your using your product product product will also change.

will also change. will also change. They will get more They will get more They will get more complicated, right? complicated, right? complicated, right? Therefore, your assessments Therefore, your assessments Therefore, your assessments should be should be should be relevant and relevant and relevant and reflect exactly what reflect exactly what reflect exactly what the customer the customer the customer is trying to do is trying to do is trying to do right now. And now right now. And now right now. And now all you need to all you need to all you need to do is simply do is simply do is simply build a cycle, build a cycle, build a cycle, release important release important release important updates, and then you updates, and then you updates, and then you will gain the trust of will gain the trust of will gain the trust of customers. That's all I customers. That's all I customers. That's all I wanted to say wanted to say wanted to say today. Thank you for today. Thank you for today. Thank you for joining. I joining. I joining. I will remain at our will remain at our will remain at our booth for further booth for further booth for further discussions, but for now, discussions, but for now, enjoy the enjoy the enjoy the conference.