The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI

Thank you to everyone who Thank you to everyone who came. I don't even came. I don't even came. I don't even know whether to be happy or know whether to be happy or know whether to be happy or sad that so many sad that so many sad that so many people are suffering from people are suffering from people are suffering from token bills. token bills. token bills. The hall is full. Yes, The hall is full. Yes, The hall is full. Yes, today we today we today we will talk about will talk about will talk about tokenomics. So, tokenomics. So, tokenomics. So, a little background a little background a little background to start with, okay? I'm to start with, okay? I'm to start with, okay? I'm Eugene from Felis AI. I am the CEO and Eugene from Felis AI. I am the CEO and Eugene from Felis AI. I am the CEO and founder of this founder of this founder of this company. I am also company. I am also company. I am also part of the open source part of the open source part of the open source Arabic AI project team Arabic AI project team Arabic AI project team , the , the , the first AI model under first AI model under first AI model under the auspices of the Linux Foundation. And we the auspices of the Linux Foundation. And we the auspices of the Linux Foundation. And we are working on the next generation AI are working on the next generation AI architecture architecture . This is a little about my . This is a little about my academic experience. academic experience. academic experience. My Twitter nickname is PicoCreator. My Twitter nickname is PicoCreator. My Twitter nickname is PicoCreator. But, more importantly, But, more importantly, But, more importantly, right? A little about Felis right? A little about Felis right? A little about Felis AI: our mission is to AI: our mission is to make AI accessible make AI accessible make AI accessible to everyone, in to everyone, in to everyone, in every language, dialect, every language, dialect, every language, dialect, and country. We do and country. We do and country. We do this by creating the this by creating the this by creating the world's largest world's largest world's largest open open open model platform, with model platform, with model platform, with over 40,000 models. over 40,000 models. over 40,000 models. This includes This includes This includes specialized specialized specialized pre-trained models, such as pre-trained models, such as Sunflower—Uganda’s first language Sunflower—Uganda’s first language Sunflower—Uganda’s first language model model model built specifically built specifically built specifically for their language—and for their language—and for their language—and many other many other many other niche niche niche solutions. So we're solutions. So we're solutions. So we're not just talking about not just talking about not just talking about the biggest the biggest the biggest open source models, but also open source models, but also open source models, but also specialized specialized specialized language models or language models or language models or specific specific specific use cases.

use cases. use cases. I will also touch on some of them I will also touch on some of them I will also touch on some of them . When we . When we . When we talk about talk about talk about scaling scaling scaling model diversity model diversity , Felis AI also , Felis AI also , Felis AI also serves customers serves customers serves customers at large scale. at large scale. at large scale. Some of our Some of our Some of our largest clients largest clients largest clients process literally process literally process literally 2 trillion tokens 2 trillion tokens 2 trillion tokens per day. They per day. They per day. They service this service this service this for about for about for about $0.3 per million $0.3 per million $0.3 per million tokens, or tokens, or tokens, or $60k per day. They $60k per day. They $60k per day. They do this do this do this on about 512 GPUs from AMD, model on about 512 GPUs from AMD, model on about 512 GPUs from AMD, model MI25. For this MI25. For this MI25. For this particular client, particular client, particular client, it is a private 400 it is a private 400 billion MOE model. And billion MOE model. And billion MOE model. And they deliver they deliver Opus 4.8-class performance. That is, this is a Opus 4.8-class performance. That is, this is a huge number of huge number of huge number of parallel requests parallel requests parallel requests and a large volume of and a large volume of and a large volume of tokens. This is what we mean tokens. This is what we mean tokens. This is what we mean by by by scale. If scale. If scale. If you look at it, $0.03 you look at it, $0.03 you look at it, $0.03 per million tokens per million tokens per million tokens at 4.8 is not something at 4.8 is not something at 4.8 is not something you can get in the you can get in the you can get in the public cloud market. public cloud market. public cloud market. These are very These are very These are very specialized specialized specialized settings settings settings designed to designed to designed to work at the scale work at the scale work at the scale we're talking about. we're talking about. This brings us to This brings us to two important two important two important caveats, which I will caveats, which I will caveats, which I will begin with. First: we begin with. First: we begin with. First: we sell inference.

sell inference. sell inference. Therefore, everything I cover here Therefore, everything I cover here will apply to any provider. There is any provider. There is any provider. There is nothing nothing nothing specific specific specific to Felis here. But, for the sake of to Felis here. But, for the sake of to Felis here. But, for the sake of clarity, we are selling clarity, we are selling clarity, we are selling inference. So this is inference. So this is inference. So this is based on our based on our based on our data and our data and our data and our experience accordingly. experience accordingly. experience accordingly. OK? One more thing: OK? One more thing: OK? One more thing: this workshop covers this workshop covers this workshop covers many aspects of many aspects of many aspects of optimizing at optimizing at optimizing at scale, and it scale, and it scale, and it probably won’t be right for probably won’t be right for probably won’t be right for you if you you if you you if you spend less than spend less than spend less than $10k per $10k per $10k per month in your month in your month in your organization. My organization. My organization. My advice to those below advice to those below advice to those below that mark—and this is an that mark—and this is an that mark—and this is an arbitrary figure— arbitrary figure— is that is that is that if you are spending if you are spending if you are spending less, you are better off less, you are better off less, you are better off focusing on focusing on focusing on finding compelling finding compelling use cases within your use cases within your company than company than company than optimizing optimizing optimizing token accounts. Because a token accounts. Because a token accounts. Because a compelling compelling compelling use case can use case can use case can bring your bring your bring your organization significantly organization significantly organization significantly more than 10k per more than 10k per more than 10k per month. And that's exactly what month. And that's exactly what month. And that's exactly what you should you should you should focus on, but that's focus on, but that's focus on, but that's not the topic of my not the topic of my not the topic of my talk. So, in talk. So, in talk. So, in this seminar we will this seminar we will this seminar we will look at four look at four look at four main steps: main steps: main steps: measurement, sizing measurement, sizing , how to procure , how to procure capacity, and capacity, and capacity, and process validation.

process validation. So, measurements. So, measurements. Yes. You will be Yes. You will be Yes. You will be surprised how surprised how surprised how important this is. How many of important this is. How many of important this is. How many of you know your you know your you know your total total total token spending? Can you token spending? Can you token spending? Can you raise your hands? Wow, raise your hands? Wow, raise your hands? Wow, only half the audience only half the audience only half the audience raised their hands. OK. raised their hands. OK. raised their hands. OK. How many of you can How many of you can How many of you can measure this at the measure this at the measure this at the department level? Those who department level? Those who department level? Those who have raised their hands, keep have raised their hands, keep have raised their hands, keep them raised. I see that them raised. I see that them raised. I see that half of them have given up half of them have given up half of them have given up . Yes. And the indicator is . Yes. And the indicator is . Yes. And the indicator is getting lower and lower at the getting lower and lower at the getting lower and lower at the team level and at the level of team level and at the level of team level and at the level of specific tasks. specific tasks. specific tasks. At least 30-40% of our At least 30-40% of our At least 30-40% of our clients who clients who clients who contact us contact us contact us initially had a single initially had a single initially had a single shared key that shared key that shared key that was used by everyone in the was used by everyone in the was used by everyone in the organization. And it organization. And it organization. And it was like: how do we was like: how do we was like: how do we even measure this? even measure this? even measure this? Where should we start? Where should we start? Where should we start? So, it doesn't depend So, it doesn't depend So, it doesn't depend on a specific on a specific on a specific provider: you provider: you provider: you need to separate need to separate need to separate the keys, allocate them the keys, allocate them the keys, allocate them by departments and by departments and by departments and main tasks main tasks , so you know where , so you know where , so you know where your tokens are being spent your tokens are being spent your tokens are being spent . And then it will . And then it will . And then it will help you help you help you optimize your optimize your optimize your workload. Yes, workload. Yes, workload. Yes, creating API keys creating API keys creating API keys shouldn't be a shouldn't be a shouldn't be a difficult task.

difficult task. difficult task. Once you Once you Once you know this, you can know this, you can know this, you can start start start optimizing optimizing optimizing resource sizes accordingly. For example, resource sizes accordingly. For example, resource sizes accordingly. For example, which team which team is spending money on which workload? What is spending money on which workload? What exactly did exactly did exactly did the person expect? Some teams the person expect? Some teams the person expect? Some teams deserve a deserve a deserve a certain number of certain number of certain number of tokens per second. And tokens per second. And tokens per second. And background agents background agents background agents running at midnight running at midnight running at midnight probably don't probably don't probably don't deserve the deserve the deserve the tokens per tokens per tokens per second rate you're second rate you're second rate you're paying for. You don't need to paying for. You don't need to paying for. You don't need to pay for the Ultra plan pay for the Ultra plan pay for the Ultra plan for a background agent. for a background agent. for a background agent. This is simply absurd. This is simply absurd. This is simply absurd. Um, and with that you Um, and with that you Um, and with that you can also better can also better can also better analyze analyze analyze usage to more usage to more usage to more effectively effectively effectively plan the dedicated plan the dedicated plan the dedicated capacities that you capacities that you capacities that you can purchase, can purchase, can purchase, for both open and for both open and for both open and closed models. closed models. closed models. Yes. You can also Yes. You can also Yes. You can also check which check which check which team is spending team is spending team is spending the most tokens and the most tokens and the most tokens and whether they are whether they are whether they are productive. I will productive. I will productive. I will also touch on this also touch on this also touch on this issue. One issue. One issue. One thing to thing to thing to know about AI know about AI workloads, and workloads, and workloads, and one you may have already one you may have already one you may have already encountered, is encountered, is encountered, is that there are two that there are two that there are two main directions with main directions with main directions with diametrically diametrically diametrically opposed opposed opposed economics. So, there are economics. So, there are economics. So, there are agent loads agent loads agent loads and, of course, intermediate and, of course, intermediate and, of course, intermediate options, where these are options, where these are options, where these are usually usually usually long-term long-term long-term multi-pass agents. multi-pass agents. multi-pass agents. Coding agents are a Coding agents are a Coding agents are a prime example of prime example of prime example of this approach. Some this approach. Some this approach. Some developers may developers may developers may run coding run coding run coding agents literally agents literally agents literally overnight for overnight for overnight for long periods of time with long periods of time with long periods of time with multiple multiple multiple compactions. And these compactions. And these compactions. And these things maximize the things maximize the things maximize the use of use of use of context.

context. Process automation is the other Process automation is the other extreme. Think extreme. Think extreme. Think call centers, call centers, email routing, email routing, meeting notes, etc. meeting notes, etc. meeting notes, etc. They usually They usually They usually use use use much less much less much less context, although there are context, although there are context, although there are many examples that many examples that many examples that contradict this contradict this contradict this statement. But this is a statement. But this is a statement. But this is a very very very process-oriented, process-oriented, process-oriented, task-isolated approach, task-isolated approach, task-isolated approach, essentially a game of essentially a game of essentially a game of volume. So what do I volume. So what do I volume. So what do I mean by mean by mean by more more more usage? And this is usage? And this is usage? And this is probably one of the probably one of the probably one of the first big first big first big cost explosions that cost explosions that many organizations face. many organizations face. Particularly when they are Particularly when they are Particularly when they are exposed to the hype exposed to the hype exposed to the hype around an open around an open around an open cloud environment cloud environment cloud environment or one of the cloud or one of the cloud or one of the cloud clones. For example, we clones. For example, we clones. For example, we analyzed one analyzed one analyzed one of the teams we of the teams we of the teams we support, they support, they support, they only had five only had five only had five developers, and they were developers, and they were running an average of eight running an average of eight agents at a time for one of the agents at a time for one of the agents at a time for one of the companies we companies we companies we work with. Previously, they work with. Previously, they work with. Previously, they used the Opus used the Opus used the Opus 4.8 API and spent 4.8 API and spent 4.8 API and spent approximately 150 thousand per approximately 150 thousand per approximately 150 thousand per month on the ongoing month on the ongoing month on the ongoing operation of these agents. operation of these agents. operation of these agents. Some of them are Some of them are Some of them are agents that they agents that they agents that they use to use to use to write code, write code, write code, some are those that some are those that some are those that turn on at night.

turn on at night. turn on at night. They were a little more They were a little more They were a little more advanced. They already advanced. They already advanced. They already knew that it was worth knew that it was worth knew that it was worth running the task running the task running the task at night, when no one was at night, when no one was at night, when no one was working. But, in fact, working. But, in fact, working. But, in fact, they were spending 116 they were spending 116 they were spending 116 billion tokens per billion tokens per billion tokens per month. And we month. And we month. And we helped them migrate helped them migrate helped them migrate to a GRM 5.3 instance. to a GRM 5.3 instance. to a GRM 5.3 instance. Previously it was version Previously it was version Previously it was version 5.2. They had already upgraded 5.2. They had already upgraded 5.2. They had already upgraded to 5.3, and it was a single to 5.3, and it was a single to 5.3, and it was a single node that cost node that cost node that cost less than 10k per less than 10k per less than 10k per month. And they didn't month. And they didn't month. And they didn't experience any disruptions experience any disruptions experience any disruptions or negative or negative or negative impacts. So understanding impacts. So understanding impacts. So understanding how your how your how your team works allows you team works allows you team works allows you to make appropriate to make appropriate to make appropriate decisions. And, as I've already decisions. And, as I've already decisions. And, as I've already emphasized, knowing emphasized, knowing emphasized, knowing workflows workflows workflows also allows you also allows you also allows you to divide tasks. to divide tasks. to divide tasks. For example, For example, For example, they used to run everything they used to run everything they used to run everything during the day, but during the day, but during the day, but started scheduling started scheduling started scheduling some some some background agents to run while the background agents to run while the background agents to run while the team was sleeping, and that team was sleeping, and that team was sleeping, and that runs at night. So, this runs at night. So, this runs at night. So, this allows them allows them to use to use the equipment more efficiently. As I the equipment more efficiently. As I the equipment more efficiently. As I briefly mentioned, the other briefly mentioned, the other briefly mentioned, the other side of side of process automation is purely a game process automation is purely a game of volume. of volume. Extremely clearly Extremely clearly defined. I can defined. I can defined. I can give more give more give more specific examples specific examples specific examples off the record, but off the record, but off the record, but imagine millions imagine millions imagine millions or even tens of or even tens of or even tens of millions of operations millions of operations millions of operations per day: decisions in call per day: decisions in call centers, centers, centers, lines of code checked, lines of code checked, emails and emails and messages filtered.

messages filtered. messages filtered. For example, one of the For example, one of the For example, one of the telecoms we telecoms we telecoms we work with easily work with easily work with easily reaches a peak of 10 reaches a peak of 10 reaches a peak of 10 million million million messages per messages per messages per minute. And this should come as no minute. And this should come as no surprise. "Happy New surprise. "Happy New Year" is the worst. Yes Year" is the worst. Yes . They filter . They filter . They filter messages because they messages because they messages because they provide a provide a provide a safe phone service safe phone service safe phone service for children. They for children. They for children. They needed AI to needed AI to needed AI to scan content for scan content for profanity, profanity, sexual sexual sexual overtones, etc. This is not a overtones, etc. This is not a overtones, etc. This is not a long context, but a long context, but a long context, but a game of volume. Other game of volume. Other game of volume. Other things are things are things are creation and creation and creation and tracking processes, tracking processes, tracking processes, approval processes, approval processes, approval processes, document processing. document processing. document processing. All of this is a game of All of this is a game of All of this is a game of scale for many scale for many scale for many organizations. I organizations. I organizations. I suspect that for suspect that for suspect that for financial financial financial institutions here in New institutions here in New York, a lot of OCR, York, a lot of OCR, York, a lot of OCR, documentation, approvals, documentation, approvals, documentation, approvals, and and and transaction transaction transaction risk assessments fall risk assessments fall risk assessments fall into this category. into this category. OK. So, yes. If you're not already If you're not already tracking token spending tracking token spending tracking token spending and usage and usage and usage , we're , we're , we're opening up access opening up access opening up access to the to the Tokenomics Tracking Tool—it's a Tokenomics Tracking Tool—it's a pretty handy pretty handy pretty handy way to split way to split way to split your API keys and your API keys and your API keys and track spending for track spending for track spending for each of them, a each of them, a each of them, a separate topic. Yes. If separate topic. Yes. If separate topic. Yes. If you want to update you want to update you want to update the information, feel the information, feel the information, feel free to free to free to scan this QR scan this QR code. Also, if you code. Also, if you code. Also, if you would like to receive would like to receive would like to receive slides for this slides for this slides for this talk or talk or talk or find out when find out when find out when the recording from AI Engineering will be released, you the recording from AI Engineering will be released, you the recording from AI Engineering will be released, you can fill out can fill out can fill out the form at the link.

the form at the link. the form at the link. OK. OK. OK. OK. OK. OK. OK. So, yes. It OK. So, yes. It OK. So, yes. It was about measurement. was about measurement. was about measurement. Um, yes. If you don't Um, yes. If you don't Um, yes. If you don't measure results measure results , you can , you can , you can ignore everything else ignore everything else ignore everything else I say and start I say and start I say and start with measurement, with measurement, with measurement, because measurement is where the biggest because measurement is where the biggest because measurement is where the biggest return on investment (ROI) comes from return on investment (ROI) comes from return on investment (ROI) comes from . But once . But once . But once you've you've you've measured everything, the measured everything, the measured everything, the question arises: how to properly question arises: how to properly question arises: how to properly optimize these optimize these optimize these metrics? And this is where metrics? And this is where metrics? And this is where things get more complicated. things get more complicated. In general, you know, one In general, you know, one of the things we of the things we of the things we start to see as start to see as start to see as AI AI AI is implemented across the is implemented across the organization is this organization is this tendency that because tendency that because tendency that because AI is this AI is this AI is this shiny new shiny new shiny new hammer, we hammer, we hammer, we try try try to apply it to to apply it to to apply it to everything. Especially in everything. Especially in everything. Especially in innovation innovation innovation labs, we labs, we labs, we just try to just try to just try to apply it to apply it to apply it to every little thing. And we are every little thing. And we are every little thing. And we are prone to, and I've seen prone to, and I've seen prone to, and I've seen organizations organizations organizations literally literally literally apply AI to the apply AI to the apply AI to the truly truly truly most absurd most absurd most absurd things. So, there are things. So, there are things. So, there are many models out there, and many models out there, and many models out there, and many models are very many models are very many models are very expensive, so expensive, so expensive, so figuring out what to do figuring out what to do figuring out what to do and where they fit is an and where they fit is an and where they fit is an important step in the important step in the important step in the process. I know process. I know process. I know open models, I mean open models, I mean open models, I mean many of many of many of you have probably heard of you have probably heard of you have probably heard of open models, and that's open models, and that's open models, and that's our line of business.

our line of business. our line of business. Yes. And so when Yes. And so when Yes. And so when many companies many companies many companies approach us, approach us, approach us, for example, to find for example, to find for example, to find open models that open models that open models that fit their fit their use cases and use cases and can replace can replace can replace existing ones, we usually existing ones, we usually existing ones, we usually guide them guide them guide them according to their according to their according to their needs. These are general needs. These are general needs. These are general rules that rules that will likely be updated will likely be updated monthly, as monthly, as monthly, as open source models open source models open source models are released every are released every are released every few months, the same goes for few months, the same goes for closed closed source models. Everything source models. Everything source models. Everything is changing quickly. But the thing that's is changing quickly. But the thing that's is changing quickly. But the thing that's really really really exciting about this year is that exciting about this year is that exciting about this year is that things things things are starting to change. are starting to change. are starting to change. As in the previous As in the previous As in the previous year, the best year, the best year, the best closed-source model closed-source model closed-source model was significantly better than the was significantly better than the was significantly better than the best best best open-source model. open-source model. open-source model. Almost. But as Almost. But as Almost. But as this this this year approaches, we are starting to year approaches, we are starting to year approaches, we are starting to see the gap see the gap see the gap narrowing. We are narrowing. We are narrowing. We are starting to see starting to see starting to see situations where some situations where some situations where some open models are already open models are already open models are already on par or even on par or even on par or even surpassing them. surpassing them. surpassing them. For example, before the release of the For example, before the release of the For example, before the release of the Opus 5.5 update, Kimi K3 Opus 5.5 update, Kimi K3 Opus 5.5 update, Kimi K3 outperformed Opus in outperformed Opus in user interface design in some user interface design in some UI tests. And these are fairly UI tests. And these are fairly UI tests. And these are fairly public benchmarks. In public benchmarks. In public benchmarks. In client tests, client tests, client tests, we have already started to we have already started to we have already started to see see see open source models open source models open source models winning in very winning in very winning in very specialized specialized specialized benchmarks for benchmarks for benchmarks for narrow tasks. One narrow tasks. One narrow tasks. One example example example worth highlighting is the worth highlighting is the worth highlighting is the step one five model. It was step one five model. It was step one five model. It was able to achieve able to achieve able to achieve quality close to quality close to quality close to Opus, despite being Opus, despite being Opus, despite being 10x 10x 10x smaller, in financial smaller, in financial smaller, in financial benchmarks and benchmarks and benchmarks and financial financial financial use cases. Is use cases. Is use cases. Is it the best it the best it the best model for model for model for coding? No, but in coding? No, but in coding? No, but in this particular this particular this particular case she is case she is case she is extremely strong extremely strong extremely strong and competent. So, and competent. So, and competent. So, the landscape the landscape the landscape is slowly changing and now

is slowly changing and now is slowly changing and now you can have a direct you can have a direct you can have a direct replacement for many replacement for many use cases that is use cases that is sometimes even sometimes even sometimes even 10x cheaper if 10x cheaper if 10x cheaper if you know your you know your you know your workload workload workload and can and can and can implement it accordingly. Special implement it accordingly. Special implement it accordingly. Special thanks also to DeepSeek-V 4- thanks also to DeepSeek-V 4- Flash. For some reason, Flash. For some reason, Flash. For some reason, this model is this model is this model is extremely extremely extremely popular for popular for any task, any task, any task, as many as many as many companies find it easy companies find it easy companies find it easy to manage to manage to manage with prompts. with prompts. with prompts. Well, the rest of the models Well, the rest of the models Well, the rest of the models I listed are I listed are I listed are kind of kind of kind of quality standards in quality standards in quality standards in their categories. their categories. their categories. DeepSeek-V 4-Flash seems to be DeepSeek-V 4-Flash seems to be DeepSeek-V 4-Flash seems to be quite popular quite popular quite popular regardless of the regardless of the regardless of the situation. Yes. Although situation. Yes. Although situation. Yes. Although test scores test scores test scores don't seem to don't seem to don't seem to explain why it's explain why it's explain why it's so popular. Yes. so popular. Yes. so popular. Yes. Yes. Another point Yes. Another point Yes. Another point worth worth worth paying attention to is that paying attention to is that recently, recently, recently, just a few just a few just a few weeks ago, a weeks ago, a weeks ago, a new class of AI models new class of AI models new class of AI models called " called " classifiers" appeared, classifiers" appeared, classifiers" appeared, namely "one-shot" namely "one-shot" namely "one-shot" classifiers, better classifiers, better classifiers, better known as JAFs. known as JAFs. known as JAFs. Open source projects Open source projects Open source projects have already begun have already begun have already begun to develop actively, to develop actively, to develop actively, for example, Simple JAF. These are for example, Simple JAF. These are for example, Simple JAF. These are models that models that models that focus entirely on focus entirely on focus entirely on decision-making and decision-making and decision-making and task classification task classification . The main advantage . The main advantage . The main advantage is that, on a is that, on a is that, on a purely technical purely technical purely technical level, they are usually level, they are usually level, they are usually 10 times cheaper than 10 times cheaper than 10 times cheaper than existing models, and existing models, and existing models, and often faster. The often faster. The often faster. The downside is that they downside is that they downside is that they cannot cannot cannot maintain a dialogue.

maintain a dialogue. maintain a dialogue. So these are things that are So these are things that are So these are things that are changing and changing and changing and happening, and you happening, and you happening, and you should start studying should start studying should start studying them to understand how to them to understand how to them to understand how to improve your improve your improve your token economy. token economy. Yes. But returning Yes. But returning to the issue of to the issue of to the issue of token economics, there are two token economics, there are two token economics, there are two main directions, and it is main directions, and it is main directions, and it is important to separate important to separate important to separate and track them and track them and track them separately. Regarding agents, separately. Regarding agents, separately. Regarding agents, we are talking about we are talking about we are talking about long-term processes long-term processes long-term processes with a lot of context, with a lot of context, with a lot of context, as I said before. as I said before. You should understand You should understand that such a load that such a load that such a load usually usually usually depends heavily on the cache, depends heavily on the cache, depends heavily on the cache, on the amount of input on the amount of input on the amount of input data, and only a data, and only a data, and only a small proportion is small proportion is small proportion is accounted for by the output accounted for by the output accounted for by the output data. We see that in data. We see that in agent usage scenarios, agent usage scenarios, only 1% of requests only 1% of requests only 1% of requests are for source are for source are for source tokens. This is quite tokens. This is quite tokens. This is quite logical, because the logical, because the logical, because the process consists of process consists of process consists of many many many repetitive steps. But it repetitive steps. But it repetitive steps. But it also determines also determines also determines the value of your the value of your the value of your tokens, because if you tokens, because if you tokens, because if you looked at the looked at the looked at the token prices... token prices... token prices... Deployment cost, right? Deployment cost, right?

The output may The output may have the largest number, have the largest number, have the largest number, but even if the but even if the but even if the input or cache is input or cache is input or cache is smaller, in this smaller, in this smaller, in this scenario it is scenario it is scenario it is constantly constantly constantly repeated. Yes. This is repeated. Yes. This is repeated. Yes. This is not a coding not a coding agent workshop, but it is one agent workshop, but it is one agent workshop, but it is one of the most common of the most common of the most common patterns we patterns we patterns we see in many see in many see in many organizations. organizations. Typically, when they Typically, when they try to try to try to scale scale scale AI AI agent deployments across agent deployments across agent deployments across development teams... They development teams... They development teams... They assign the best assign the best assign the best agent to plan, the agent to plan, the agent to plan, the faster and cheaper faster and cheaper faster and cheaper agent to implement, and agent to implement, and agent to implement, and then maybe the then maybe the then maybe the best agent again to best agent again to validate. There is validate. There is validate. There is a good reason why a good reason why a good reason why this distribution is very this distribution is very this distribution is very common among all common among all common among all agent frameworks agent frameworks : you get the : you get the : you get the best of both best of both best of both worlds, as the implementation phase is worlds, as the implementation phase is usually very usually very demanding on demanding on demanding on tokens. Many tokens. Many tokens. Many processes processes processes are repeated back and are repeated back and forth, such as forth, such as forth, such as editing a file, editing a file, editing a file, testing, testing, testing, checking the checking the checking the results, results, results, repeating over and over again repeating over and over again repeating over and over again . And most of . And most of . And most of these agents, especially these agents, especially these agents, especially Gemini 1.5 Flash, are quite Gemini 1.5 Flash, are quite Gemini 1.5 Flash, are quite capable of performing capable of performing capable of performing tasks at the same tasks at the same tasks at the same quality level as more quality level as more quality level as more powerful agents, powerful agents, powerful agents, such as Astra or Opus, such as Astra or Opus, such as Astra or Opus, if these agents take over the if these agents take over the if these agents take over the planning.

planning. We have worked with We have worked with organizations where organizations where organizations where planning and planning and planning and verification is performed by the verification is performed by the verification is performed by the most powerful model, most powerful model, most powerful model, and the executor is, and the executor is, and the executor is, for example, Gemini 1.5 Flash for example, Gemini 1.5 Flash for example, Gemini 1.5 Flash or GPT-4o mini. Yes. One or GPT-4o mini. Yes. One or GPT-4o mini. Yes. One observation observation observation we made about we made about we made about this approach this approach this approach is that, is that, is that, unfortunately, despite our unfortunately, despite our unfortunately, despite our hopes, no one hopes, no one hopes, no one actually changes actually changes actually changes the model just to the model just to the model just to save the save the save the company money. People are company money. People are company money. People are generally too lazy generally too lazy generally too lazy to switch models. to switch models. It is necessary to create It is necessary to create incentives for such a incentives for such a incentives for such a switch. Or, switch. Or, switch. Or, alternatively, if alternatively, if alternatively, if your agent your agent your agent framework is of sufficient framework is of sufficient framework is of sufficient quality, quality, quality, such as MemGPT, it such as MemGPT, it such as MemGPT, it can automate can automate can automate this process. It this process. It this process. It uses the weaker uses the weaker uses the weaker model as a baseline and model as a baseline and model as a baseline and then engages the then engages the then engages the more powerful model to more powerful model to more powerful model to automatically automatically automatically switch when switch when switch when better better better expertise is needed. This expertise is needed. This expertise is needed. This helps reduce helps reduce helps reduce friction and improve friction and improve friction and improve token economics token economics token economics at scale at scale at scale for users. The for users. The basic rule that basic rule that we now give to our we now give to our we now give to our customers after customers after customers after repeated repeated repeated experience is experience is experience is that you need a that you need a that you need a smarter model as a smarter model as a smarter model as a base model, but at the same time the base model, but at the same time the base model, but at the same time the faster and cheaper faster and cheaper faster and cheaper model should model should model should work even faster.

work even faster. This naturally creates the This naturally creates the necessary motivation. necessary motivation. necessary motivation. People want People want People want to get their to get their to get their work done quickly. If work done quickly. If work done quickly. If a model is a model is a model is smart enough to smart enough to smart enough to handle handle handle the task and is the task and is the task and is cheaper, they cheaper, they cheaper, they will will will use it use it use it simply because it is simply because it is simply because it is faster, not because it is faster, not because it is faster, not because it is cheaper. We had cheaper. We had cheaper. We had teams where both teams where both teams where both models, the smarter one and the models, the smarter one and the models, the smarter one and the cheaper one, were running at the cheaper one, were running at the cheaper one, were running at the same same same token per second rate, and token per second rate, and token per second rate, and as long as they had as long as they had as long as they had excess excess excess token budget, no one token budget, no one token budget, no one switched. On the other switched. On the other switched. On the other hand, we've seen hand, we've seen hand, we've seen teams go teams go teams go to the other extreme: to the other extreme: to the other extreme: they set a they set a budget so low that when budget so low that when using a using a using a smarter model, smarter model, smarter model, they would spend it all in they would spend it all in they would spend it all in one day. Yes, then one day. Yes, then one day. Yes, then everyone just everyone just everyone just switches to the faster switches to the faster switches to the faster model because it model because it model because it fits their fits their fits their budget. It's almost as budget. It's almost as budget. It's almost as if a smarter if a smarter if a smarter model didn't exist. model didn't exist. model didn't exist. So be mindful of So be mindful of So be mindful of your your your team dynamics in terms of how team dynamics in terms of how team dynamics in terms of how they switch. they switch. they switch. The best compromise The best compromise is speed, because is speed, because is speed, because people naturally strive for it people naturally strive for it people naturally strive for it . So they . So they . So they will simply choose the will simply choose the will simply choose the faster and cheaper faster and cheaper faster and cheaper option. Yes.

option. Yes. option. Yes. I remind you once again: measure, and I remind you once again: measure, and I remind you once again: measure, and measure separately for measure separately for measure separately for each each each token key. But this token key. But this token key. But this goes far beyond goes far beyond goes far beyond simple simple simple measurement. measurement. measurement. For example, CI/CD is a For example, CI/CD is a For example, CI/CD is a very common use case for very common use case for processes running processes running in the background. We helped in the background. We helped in the background. We helped teams set up a teams set up a teams set up a higher higher higher token generation rate token generation rate token generation rate for for human users, but a much human users, but a much human users, but a much slower rate for CI slower rate for CI /CD pipelines, which /CD pipelines, which /CD pipelines, which allows for significant allows for significant allows for significant savings. And it's savings. And it's savings. And it's simply a matter of simply a matter of simply a matter of what priorities you what priorities you what priorities you set for each set for each set for each task. Yes. You task. Yes. You task. Yes. You can control can control can control this at the individual this at the individual this at the individual key level. Another key level. Another key level. Another important conclusion important conclusion important conclusion we made we made we made while deploying while deploying while deploying team solutions team solutions team solutions is the following. is the following. For example, many For example, many of you have had the happy of you have had the happy of you have had the happy or unhappy experience or unhappy experience or unhappy experience of a CEO saying, "Everyone of a CEO saying, "Everyone of a CEO saying, "Everyone should implement AI should implement AI should implement AI now, and everyone should now, and everyone should now, and everyone should get a second get a second get a second brain." Yes. Many brain." Yes. Many brain." Yes. Many organizations organizations organizations have already started this process. In have already started this process. In have already started this process. In some cases, such as the some cases, such as the UAE, this has become a UAE, this has become a UAE, this has become a literal requirement of the literal requirement of the literal requirement of the law. So many law. So many law. So many departments are simply not departments are simply not departments are simply not ready to understand ready to understand ready to understand what what what the compromise is. So, a the compromise is. So, a the compromise is. So, a good rule for the good rule for the good rule for the future. Developers tend future. Developers tend future. Developers tend to understand to understand to understand the difference between models, the difference between models, the difference between models, for example, DeepSeek Flash and

for example, DeepSeek Flash and for example, DeepSeek Flash and GR553. Your finance GR553. Your finance GR553. Your finance department likely department likely department likely has no idea about this at all has no idea about this at all has no idea about this at all . Your . Your . Your support department has no support department has no support department has no clue either. They clue either. They clue either. They will simply choose the first will simply choose the first will simply choose the first best model and will best model and will best model and will probably stick probably stick probably stick with it. Therefore, our with it. Therefore, our with it. Therefore, our general rule is: general rule is: name department models by their name department models by their functions, not by the functions, not by the functions, not by the names of the names of the names of the models themselves. This gives you models themselves. This gives you models themselves. This gives you much more much more much more control when control when control when replacing them. As replacing them. As replacing them. As mentioned earlier: mentioned earlier: mentioned earlier: fast vs. fast vs. fast vs. smart, or in smart, or in smart, or in some cases, some cases, some cases, financial agent or financial agent or spreadsheet agent. These are spreadsheet agent. These are hyper-specialized hyper-specialized hyper-specialized agents who have been agents who have been agents who have been further trained for the further trained for the further trained for the appropriate tasks. appropriate tasks. appropriate tasks. And it also gives you And it also gives you And it also gives you significantly more significantly more significantly more control. This allows control. This allows control. This allows you to easily replace you to easily replace you to easily replace the model. This allows you the model. This allows you the model. This allows you to split traffic and to split traffic and to split traffic and test it test it test it without having to without having to without having to say, "Hey, say, "Hey, say, "Hey, users, users, users, please try this." please try this." please try this." The downside is that The downside is that The downside is that users users users can sometimes complain can sometimes complain can sometimes complain when you do this and when you do this and when you do this and make a make a make a mistake. But having that kind of mistake. But having that kind of mistake. But having that kind of control is control is control is good. Another pattern good. Another pattern good. Another pattern we see, we see, we see, even if you even if you even if you do this, is perhaps do this, is perhaps do this, is perhaps some kind of previous some kind of previous some kind of previous version. So, this is a version. So, this is a version. So, this is a variation of the same thing, variation of the same thing, variation of the same thing, but it gives you but it gives you but it gives you control without control without control without burdening burdening burdening the organization with unnecessary the organization with unnecessary the organization with unnecessary communication. And this is communication. And this is communication. And this is extremely important extremely important extremely important because, for example, because, for example, some of the teams some of the teams we work with, we've we work with, we've we work with, we've literally seen literally seen literally seen participants launch participants launch participants launch Llama 2 models as we're Llama 2 models as we're Llama 2 models as we're talking about this talking about this talking about this year because no one year because no one year because no one changed the settings.

changed the settings. Yeah, so you really Yeah, so you really need to set need to set need to set this in place. Yes. this in place. Yes. OK. Process OK. Process track. Process track. track. Process track. track. Process track. Much of what I Much of what I Much of what I mentioned earlier is not mentioned earlier is not mentioned earlier is not entirely applicable to entirely applicable to entirely applicable to process tracking, process tracking, process tracking, as it is as it is as it is usually heavily usually heavily usually heavily integrated into integrated into integrated into applications and applications and applications and workflows, respectively. And workflows, respectively. And workflows, respectively. And this is simply due to the this is simply due to the this is simply due to the huge volume of huge volume of huge volume of traffic. Aha. And all traffic. Aha. And all traffic. Aha. And all the teams in the process the teams in the process the teams in the process track tend to be track tend to be track tend to be much more much more much more focused on focused on focused on getting the getting the getting the default size right default size right default size right , , , because if you're processing because if you're processing because if you're processing trillions of tokens, trillions of tokens, trillions of tokens, most departments most departments most departments won't approve of a price of won't approve of a price of won't approve of a price of $10 per million $10 per million $10 per million tokens. This is simply tokens. This is simply tokens. This is simply impossible. So they impossible. So they impossible. So they start early with start early with start early with assessments, benchmarks, and assessments, benchmarks, and assessments, benchmarks, and then look for a model then look for a model then look for a model that fits the that fits the that fits the size. Yes. So, a size. Yes. So, a size. Yes. So, a common question is: How do common question is: How do common question is: How do I know if a I know if a smaller model is good enough? You don't smaller model is good enough? You don't need to...That's the need to...That's the need to...That's the wrong wrong wrong question. You just question. You just question. You just need to need to need to make sure you have make sure you have make sure you have your your your test cases, test cases, test cases, evaluations, and examples set up. And evaluations, and examples set up. And evaluations, and examples set up. And then just then just then just carry out the carry out the carry out the search process from that search process from that search process from that point on. Companies like point on. Companies like point on. Companies like ours will be happy ours will be happy ours will be happy to help you to help you to help you as we have as we have as we have over 40,000 models.

over 40,000 models. over 40,000 models. But, in the end, it's not that But, in the end, it's not that But, in the end, it's not that hard to hard to hard to download 10 download 10 download 10 models from Hugging Face and models from Hugging Face and models from Hugging Face and just try them out. just try them out. just try them out. So, after that, you So, after that, you So, after that, you just do just do just do a search, find a search, find a search, find the cheapest model the cheapest model the cheapest model that passes that passes that passes most of the tests, and most of the tests, and most of the tests, and then look at how to then look at how to then look at how to iterate and iterate and iterate and fine-tune it fine-tune it fine-tune it further. We also have a further. We also have a further. We also have a more specific more specific more specific process that I process that I process that I don't think is don't think is don't think is unique to us, and unique to us, and unique to us, and we'll look at that as well we'll look at that as well we'll look at that as well . . . For example, for one For example, for one For example, for one of the large telecoms of the large telecoms of the large telecoms that we work with, that we work with, that we work with, right? We actually right? We actually right? We actually helped their helped their helped their team reduce their team reduce their team reduce their inference costs from inference costs from inference costs from 250k to 7.6k per 250k to 7.6k per 250k to 7.6k per month by moving from an month by moving from an month by moving from an existing existing existing closed-source model, closed-source model, closed-source model, just going through just going through just going through a process like, “Okay, a process like, “Okay, a process like, “Okay, you you you use the use the use the best model first. " best model first. " best model first. " Let's see Let's see Let's see what models we can what models we can what models we can identify, then what identify, then what identify, then what models we can models we can models we can implement, and what we implement, and what we implement, and what we can further train?" And can further train?" And can further train?" And that was, that was to that was, that was to that was, that was to support support customer support usage scenarios and so on customer support usage scenarios and so on . Yes. And to . Yes. And to . Yes. And to emphasize again emphasize again emphasize again Jeff and Simple Jeff, which Jeff and Simple Jeff, which Jeff and Simple Jeff, which is an open-source project that is an open-source project that is an open-source project that we did that we did that we did that replicates this. You'd replicates this. You'd replicates this. You'd be surprised what be surprised what be surprised what percentage of your AI percentage of your AI traffic is traffic is traffic is classification classification classification traffic. After all, right?

traffic. After all, right? traffic. After all, right? Like, yes, of course. The Like, yes, of course. The customer support response is text. But even is text. But even is text. But even then, right? Even then then, right? Even then , right? A lot of teams, , right? A lot of teams, , right? A lot of teams, right? They already have right? They already have predefined text predefined text output that they output that they output that they insert into the insert into the insert into the context for the model context for the model context for the model to produce a to produce a to produce a result in result in result in most cases. And most cases. And most cases. And you could you could you could use a use a use a classifier, and that's a classifier, and that's a classifier, and that's a new shift in the AI ​​world, new shift in the AI ​​world, new shift in the AI ​​world, and that's one of the big and that's one of the big and that's one of the big levers for those kinds of levers for those kinds of levers for those kinds of processes in reducing processes in reducing processes in reducing your your your token bill. Yes. So, token bill. Yes. So, token bill. Yes. So, one example of such a one example of such a one example of such a model when it comes model when it comes model when it comes to additional education. So there are a to additional education. So there are a to additional education. So there are a lot of models and lot of models and lot of models and processes, right? processes, right? processes, right? They win at this They win at this They win at this scale when you scale when you scale when you start to train start to train start to train your own model. On your own model. On your own model. On the other hand, regarding the other hand, regarding coding models, there are coding models, there are organizations that organizations that organizations that retrain models retrain models retrain models for you, such as for you, such as Devin. Many Devin. Many Devin. Many developers usually developers usually developers usually just want the just want the just want the best model. So, best model. So, best model. So, unless you work in unless you work in unless you work in this AI field, it's quite this AI field, it's quite this AI field, it's quite difficult to keep up difficult to keep up difficult to keep up with training the best with training the best with training the best model for coding.

model for coding. But on the other hand, for But on the other hand, for your area, it may be your area, it may be your area, it may be worth retraining worth retraining worth retraining the domain model for the domain model for the domain model for your scenario, your scenario, your scenario, especially if the model is especially if the model is especially if the model is much smaller. much smaller. For example, for AT&T, For example, for AT&T, when they refined when they refined when they refined their models, they their models, they their models, they went from went from went from open class to open class to open class to 27B and even 7B models. 27B and even 7B models. 27B and even 7B models. And they published And they published And they published it. We were one of it. We were one of it. We were one of their their their launch partners, and you can launch partners, and you can launch partners, and you can try out their try out their try out their open open open telecommunications telecommunications telecommunications models, and that's actually what models, and that's actually what models, and that's actually what they they they use behind use behind use behind the scenes. And here it all the scenes. And here it all the scenes. And here it all depends on the volume. depends on the volume. depends on the volume. I think most of I think most of I think most of you understand the scale of you understand the scale of you understand the scale of AT&T in call centers and the like. AT&T in call centers and the like. AT&T in call centers and the like. Yes, this is a good Yes, this is a good Yes, this is a good example of how you example of how you example of how you retrain models retrain models retrain models for specialized for specialized use cases. And once use cases. And once you get to you get to you get to that point, other that point, other that point, other big organizations big organizations big organizations like Shopify like Shopify like Shopify are starting to do it, are starting to do it, are starting to do it, and it doesn't have to and it doesn't have to and it doesn't have to be a massive be a massive be a massive learning curve. So, learning curve. So, learning curve. So, for example, in the case for example, in the case for example, in the case of Shopify, they of Shopify, they of Shopify, they did it because Shopify did it because Shopify had its own had its own programming language for programming language for programming language for setting up its setting up its setting up its online store. And the online store. And the online store. And the existing models weren't existing models weren't existing models weren't very good at it, very good at it, very good at it, so they decided to so they decided to so they decided to train them. But AT&T train them. But AT&T train them. But AT&T also had similar also had similar also had similar examples, particularly examples, particularly examples, particularly in terms of how in terms of how in terms of how to place to place to place cell towers and cell towers and cell towers and things like that. Yes.

things like that. Yes. things like that. Yes. Very specific Very specific Very specific tasks for tasks for tasks for telecommunications. telecommunications. telecommunications. Maybe I'm not the Maybe I'm not the Maybe I'm not the expert to expert to expert to answer this answer this answer this question. But they question. But they question. But they had these examples of had these examples of had these examples of use. So use. So use. So they conducted they conducted they conducted appropriate additional appropriate additional appropriate additional training. You too training. You too training. You too may have such may have such usage scenarios. And how do usage scenarios. And how do you get started with you get started with you get started with such a model, right such a model, right such a model, right ? The good news ? The good news ? The good news is that is that is that it can now be easily it can now be easily it can now be easily done, because if, done, because if, done, because if, say, you want the say, you want the say, you want the best model best model best model that is too expensive that is too expensive that is too expensive for large-scale for large-scale for large-scale use to use to use to do the job, you do the job, you do the job, you can already get started. can already get started. So, an easy way So, an easy way to do this, right? We to do this, right? We to do this, right? We often see that you often see that you often see that you either generate your own either generate your own either generate your own data or prepare it data or prepare it data or prepare it yourself, which is difficult. yourself, which is difficult. yourself, which is difficult. For some reason, no For some reason, no For some reason, no one one one likes this. But likes this. But likes this. But another common path is another common path is to start with an expensive to start with an expensive to start with an expensive model, deploy it model, deploy it model, deploy it on a small on a small on a small scale, and then scale, and then scale, and then generate generate generate datasets. I understand that in datasets. I understand that in datasets. I understand that in some laboratories some laboratories some laboratories this is against the terms of this is against the terms of this is against the terms of service, so service, so service, so you should consult you should consult you should consult with lawyers, but there are with lawyers, but there are with lawyers, but there are laboratories that laboratories that laboratories that openly openly openly support this.

support this. support this. For example, Google Gemini, For example, Google Gemini, For example, Google Gemini, their advanced models, their advanced models, their advanced models, literally have a literally have a literally have a special API special API service for this. This allows service for this. This allows service for this. This allows you to create you to create you to create datasets for datasets for datasets for your case your case your case based on a powerful based on a powerful based on a powerful model, which can then be model, which can then be scaled down significantly, say to scaled down significantly, say to scale 7B. Additionally, scale 7B. Additionally, scale 7B. Additionally, open source models are open source models are quite suitable for quite suitable for startup and startup and startup and distillation, so you distillation, so you distillation, so you can can can use the use the use the best best best open source models. open source models. open source models. After collecting data, you After collecting data, you After collecting data, you can conduct further can conduct further can conduct further training and also training and also training and also try different try different try different model sizes to model sizes to model sizes to see what see what see what fits best. fits best. fits best. So, again, like So, again, like So, again, like in the 80/20 example, they have in the 80/20 example, they have in the 80/20 example, they have 27B and 7B models for 27B and 7B models for 27B and 7B models for different levels of the call different levels of the call center. For example, center. For example, center. For example, your regular your regular your regular calls might calls might calls might go to... go to... Wait. I don't think I Wait. I don't think I Wait. I don't think I can say what can say what can say what size size size fits where, but your fits where, but your fits where, but your standard queries standard queries standard queries could go to a could go to a could go to a much smaller much smaller much smaller model and then model and then model and then be routed be routed be routed higher up through certain higher up through certain classifier triggers, as an classifier triggers, as an example. Then this is example. Then this is example. Then this is how you configure the how you configure the how you configure the traffic flow, traffic flow, traffic flow, divide it, and divide it, and divide it, and achieve the desired achieve the desired achieve the desired level of efficiency.

level of efficiency. One of the things One of the things we typically we typically we typically help teams with help teams with help teams with during this process during this process is that even when is that even when is that even when they go through all the they go through all the they go through all the stages and completely stages and completely stages and completely migrate their migrate their migrate their traffic to the retrained traffic to the retrained traffic to the retrained model, as the model, as the model, as the best models best models best models continue continue continue to evolve, we to evolve, we to evolve, we typically advise them typically advise them typically advise them to route some of the to route some of the to route some of the traffic as shadow traffic as shadow traffic as shadow traffic to the best traffic to the best traffic to the best available model to available model to available model to continue to continue to continue to populate the populate the populate the dataset. And this allows dataset. And this allows dataset. And this allows you to stay on top of the you to stay on top of the you to stay on top of the progress of the progress of the progress of the best open best open best open models. The unit models. The unit models. The unit economics here are economics here are economics here are actually really actually really actually really important because, you know, I important because, you know, I important because, you know, I would like to remind would like to remind would like to remind people that they people that they people that they seriously seriously seriously underestimate underestimate underestimate the speed of the speed of the speed of AI development, especially for AI development, especially for AI development, especially for models that can be models that can be run much cheaper at scale. Take Take this laptop for example. this laptop for example. this laptop for example. I like I like I like to check what is the to check what is the to check what is the best model best model best model I can run on the I can run on the I can run on the most expensive MacBook Pro. most expensive MacBook Pro. And today's And today's best models, be best models, be best models, be it MoE models of size it MoE models of size it MoE models of size 31B, 32B, 27B, are already significantly 31B, 32B, 27B, are already significantly 31B, 32B, 27B, are already significantly better than GPT-4. So, better than GPT-4. So, better than GPT-4. So, essentially, this model that essentially, this model that essentially, this model that can run on a can run on a can run on a laptop is already better laptop is already better laptop is already better than the best model you than the best model you than the best model you had had had last year. Plus- last year. Plus- minus. People like to minus. People like to minus. People like to argue about argue about argue about details, but let's talk in details, but let's talk in details, but let's talk in general. And when you general. And when you general. And when you retrain it to retrain it to retrain it to your task, it your task, it your task, it means that if you means that if you means that if you can run that can run that can run that model on a laptop, model on a laptop, model on a laptop, you can run you can run you can run it on your GPU it on your GPU servers at scale servers at scale servers at scale much more efficiently

much more efficiently . Yes. This brings . Yes. This brings . Yes. This brings us to the next topic us to the next topic : buying a GPU, which is : buying a GPU, which is : buying a GPU, which is obviously becoming an increasingly obviously becoming an increasingly obviously becoming an increasingly difficult task difficult task . And this is where . And this is where companies like us come to the rescue. One of the things we've One of the things we've realized from realized from realized from many conversations with many conversations with many conversations with customers is that customers is that customers is that people are quite people are quite people are quite surprised that everything surprised that everything surprised that everything scales non- scales non- scales non- linearly. By this I linearly. By this I linearly. By this I mean that there is a mean that there is a mean that there is a huge trade-off huge trade-off huge trade-off between the number of between the number of between the number of tokens per second in a tokens per second in a tokens per second in a single request and single request and single request and the cost of the the cost of the the cost of the request itself. So, request itself. So, request itself. So, for example, if you for example, if you for example, if you don't care about cost, don't care about cost, don't care about cost, you could run the you could run the you could run the best model, let best model, let best model, let 's say the fixed 's say the fixed 's say the fixed model, on the best model, on the best model, on the best hardware, on all hardware, on all hardware, on all eight GPUs, with a eight GPUs, with a eight GPUs, with a batch size of one, and you batch size of one, and you batch size of one, and you could get, could get, could get, let's say, 100 tokens per let's say, 100 tokens per let's say, 100 tokens per second, 200 tokens per second, 200 tokens per second, 200 tokens per second. If you second. If you second. If you really want—and this is really want—and this is really want—and this is not our not our not our specialty—if specialty—if specialty—if you really need, you really need, you really need, for example, thousands of for example, thousands of for example, thousands of tokens per second, this is where tokens per second, this is where companies like Cerebras will be companies like Cerebras will be happy to offer happy to offer happy to offer you that. And it costs you that. And it costs you that. And it costs even more. So, even more. So, even more. So, there are definitely there are definitely there are definitely use cases use cases where this makes sense, but a where this makes sense, but a where this makes sense, but a lot of people lot of people lot of people fall into the trap of fall into the trap of fall into the trap of thinking, "Hey, if thinking, "Hey, if thinking, "Hey, if I have the speed, I have the speed, I have the speed, maybe it costs maybe it costs maybe it costs half the price." That's half the price." That's half the price." That's not even close not even close . It's like a rule of thumb . It's like a rule of thumb . It's like a rule of thumb that I sometimes that I sometimes that I sometimes say: every say: every say: every doubling of speed doubling of speed is a tripling of is a tripling of is a tripling of cost, and it's cost, and it's cost, and it's progressive. In this progressive. In this progressive. In this particular example particular example particular example for GM 5.3, a single H100, as you for GM 5.3, a single H100, as you for GM 5.3, a single H100, as you can see on the can see on the can see on the same server, yields 112 same server, yields 112 same server, yields 112 tokens per second tokens per second tokens per second when executing when executing when executing individual queries, but individual queries, but individual queries, but if you run 16 if you run 16 if you run 16 queries in parallel,

queries in parallel, queries in parallel, you get you get you get about half the about half the about half the speed for all speed for all speed for all queries, but queries, but queries, but literally 10x the literally 10x the literally 10x the overall overall overall throughput. And that gives you throughput. And that gives you throughput. And that gives you about 9-10 times about 9-10 times about 9-10 times lower cost just lower cost just lower cost just by changing the by changing the by changing the target target target velocity profile. So when velocity profile. So when velocity profile. So when many customers many customers many customers ask us, how do we ask us, how do we ask us, how do we deploy systems at deploy systems at deploy systems at scale? Sometimes the first scale? Sometimes the first scale? Sometimes the first question is: what quality of question is: what quality of question is: what quality of model do you need? model do you need? model do you need? And what speed in And what speed in And what speed in tokens per second do tokens per second do tokens per second do you need? And then you need? And then you need? And then we start we start we start recommending recommending recommending the number of GPUs and the the number of GPUs and the the number of GPUs and the deployment size. And this deployment size. And this deployment size. And this target of target of target of tokens per second is an tokens per second is an tokens per second is an important trade-off important trade-off . Essentially. Um, in more . Essentially. Um, in more . Essentially. Um, in more extreme extreme extreme cases, right? If you cases, right? If you cases, right? If you can actually can actually can actually run everything at run everything at run everything at 10 tokens 10 tokens 10 tokens per second—although it's per second—although it's per second—although it's rare that anyone does, rare that anyone does, rare that anyone does, there are some there are some there are some specific specific specific use cases that use cases that use cases that allow for it. You can allow for it. You can allow for it. You can get even more get even more get even more incredible total incredible total incredible total bandwidth amounts bandwidth amounts bandwidth amounts . . . We typically see customers We typically see customers We typically see customers choose speeds choose speeds choose speeds between 20 and 30, which is between 20 and 30, which is between 20 and 30, which is roughly the roughly the roughly the speed of your ChatGPT speed of your ChatGPT speed of your ChatGPT without much without much without much thought, and up to 60, which is thought, and up to 60, which is thought, and up to 60, which is something like the something like the something like the highest speed. And highest speed. And highest speed. And if they need 120 if they need 120 if they need 120 tokens, that's usually the tokens, that's usually the range. Most range. Most settle on a settle on a settle on a range of 20-30 to 60 range of 20-30 to 60 , depending on , depending on , depending on how long the how long the how long the task chain is. Yes task chain is. Yes . So, yes. I already . So, yes. I already . So, yes. I already mentioned that you mentioned that you mentioned that you typically only want to typically only want to typically only want to pay for speed pay for speed pay for speed in tokens per second in tokens per second in tokens per second when it when it when it directly directly directly interacts with interacts with interacts with the user. Whether it's the user. Whether it's the user. Whether it's in front of your in front of your in front of your developers or developers or developers or your your your customers. These are exactly the customers. These are exactly the customers. These are exactly the compromises you compromises you compromises you want to make. And even want to make. And even want to make. And even in this case, right in this case, right in this case, right ? We've seen ? We've seen ? We've seen cases where even

cases where even cases where even a slight decrease in a slight decrease in a slight decrease in settings has the same effect, right settings has the same effect, right settings has the same effect, right ? Can easily ? Can easily ? Can easily save twice save twice save twice as much money with as much money with as much money with almost no almost no almost no impact on the customer. impact on the customer. Yes. And again, background Yes. And again, background tasks: please tasks: please tasks: please don't run them at don't run them at don't run them at 120 tokens 120 tokens 120 tokens per second. This doesn't make per second. This doesn't make per second. This doesn't make any sense. Yes. any sense. Yes. So, like I said So, like I said before, before, before, measure first, okay? You will measure first, okay? You will measure first, okay? You will be able to see be able to see be able to see your traffic patterns. your traffic patterns. your traffic patterns. Many traffic patterns Many traffic patterns are inherently are inherently spike-like spike-like spike-like with some with some with some base base base load load load because someone is because someone is because someone is always running something. always running something. always running something. Especially background Especially background Especially background agents. And this is where you agents. And this is where you agents. And this is where you can also can also can also plan your plan your plan your deployments. I know deployments. I know deployments. I know that in the financial sector that in the financial sector that in the financial sector this is not always possible this is not always possible this is not always possible due to the use of due to the use of due to the use of dedicated instances. dedicated instances. But if there is a But if there is a hybrid option where hybrid option where hybrid option where dedicated instances dedicated instances dedicated instances are combined with are combined with are combined with peak peak peak loads in the loads in the loads in the cloud, it gives some cloud, it gives some cloud, it gives some flexibility. The biggest flexibility. The biggest flexibility. The biggest savings today savings today is deploying a is deploying a is deploying a dedicated instance dedicated instance dedicated instance if you know your if you know your if you know your baseline workload.

baseline workload. baseline workload. As long as you use it to its fullest As long as you use it to its fullest As long as you use it to its fullest , you , you , you get some get some get some savings. But it's savings. But it's savings. But it's much more complicated much more complicated much more complicated depending on your depending on your depending on your traffic, and many of traffic, and many of traffic, and many of these things can be these things can be these things can be tested and tested and tested and verified. Essentially, verified. Essentially, verified. Essentially, this is how any AI inference company will work with this is how any AI inference company will work with this is how any AI inference company will work with you you . This does . This does n't just apply to n't just apply to n't just apply to us. We will ask: "Do us. We will ask: "Do us. We will ask: "Do you know your you know your you know your work traffic? work traffic? work traffic? Can you provide the data?" And Can you provide the data?" And Can you provide the data?" And then we'll then we'll then we'll say, "Okay, we're say, "Okay, we're say, "Okay, we're offering this. offering this. offering this. Try switching to Try switching to Try switching to this option." And then this option." And then this option." And then we can usually we can usually we can usually show you your show you your show you your savings. In extreme savings. In extreme savings. In extreme cases with large cases with large cases with large peaks, it is better peaks, it is better peaks, it is better to limit the allocated to limit the allocated to limit the allocated capacity and capacity and direct excess traffic to the direct excess traffic to the public cloud. Different public cloud. Different public cloud. Different cases require cases require cases require different different different scaling, but scaling, but scaling, but measuring measuring measuring workloads in workloads in workloads in tokens allows you tokens allows you tokens allows you to start exploring these to start exploring these to start exploring these options, and the savings options, and the savings options, and the savings can be huge. can be huge. As in the case of As in the case of a client for whom a client for whom a client for whom we processed two we processed two we processed two trillion tokens on a trillion tokens on a trillion tokens on a 4.8 Opus class model, 4.8 Opus class model, 4.8 Opus class model, they pay less they pay less they pay less than 0.03 per million than 0.03 per million than 0.03 per million tokens.

tokens. tokens. You won't You won't You won't find this price anywhere else. Yes. So a find this price anywhere else. Yes. So a find this price anywhere else. Yes. So a simple caveat: simple caveat: simple caveat: peak loads peak loads peak loads really do really do really do matter. Eh, it's very matter. Eh, it's very matter. Eh, it's very rare for someone to have a rare for someone to have a rare for someone to have a brief burst at an brief burst at an brief burst at an extreme level when extreme level when extreme level when they're doing a workload of they're doing a workload of this magnitude every month this magnitude every month , and the rest doesn't , and the rest doesn't , and the rest doesn't matter. So matter. So matter. So there is usually some kind of there is usually some kind of there is usually some kind of basic basic basic workload. The workload. The workload. The only question is only question is only question is how how how big it is. Yes. Yes. So, big it is. Yes. Yes. So, big it is. Yes. Yes. So, lastly—check lastly—check lastly—check your processes. This has your processes. This has your processes. This has nothing to do with nothing to do with nothing to do with optimizing optimizing optimizing conclusions. Some of conclusions. Some of conclusions. Some of the slides may the slides may the slides may seem very seem very seem very critical of AI, but critical of AI, but critical of AI, but I'm generally optimistic. I I'm generally optimistic. I I'm generally optimistic. I just want to just want to just want to point this out because things point this out because things point this out because things can be taken out of can be taken out of can be taken out of context. So one context. So one context. So one of the Pandora's box issues of the Pandora's box issues of the Pandora's box issues or the or the or the problematic issues problematic issues problematic issues that we're starting to that we're starting to that we're starting to see is that see is that see is that a lot of organizations a lot of organizations a lot of organizations are asking, "Hey, I've are asking, "Hey, I've are asking, "Hey, I've increased increased increased token spending 10-100x, token spending 10-100x, token spending 10-100x, but I'm not seeing but I'm not seeing but I'm not seeing results at the same results at the same results at the same scale." We're scale." We're scale." We're starting to see starting to see starting to see reports from various reports from various reports from various organizations that organizations that organizations that AI has limited AI has limited AI has limited performance, only performance, only performance, only 6% or 7%, which is not at all the 6% or 7%, which is not at all the 6% or 7%, which is not at all the scale that scale that scale that many of us many of us many of us were promised or were promised or were promised or expected. Unfortunately, that's the case expected. Unfortunately, that's the case expected. Unfortunately, that's the case , because we're , because we're , because we're at a point now where at a point now where at a point now where everyone just takes a everyone just takes a everyone just takes a huge hammer and huge hammer and huge hammer and hits everything with it. It's hits everything with it. It's

not that not that AI isn't useful, it's AI isn't useful, it's AI isn't useful, it's just that we tend to just that we tend to just that we tend to stuff it everywhere. stuff it everywhere. And there are times And there are times when I feel like when I feel like when I feel like saying, "Can saying, "Can saying, "Can you please not you please not you please not make this another co- make this another co- make this another co- pilot?" Because there is pilot?" Because there is pilot?" Because there is simply simply simply too much of it. You too much of it. You too much of it. You need to think about need to think about need to think about how you how you how you use it. Uh, use it. Uh, use it. Uh, yes. Another big yes. Another big yes. Another big problem we are problem we are problem we are starting starting starting to realize is that while to realize is that while to realize is that while AI is being deployed on a AI is being deployed on a AI is being deployed on a larger scale and larger scale and larger scale and more frequently, and humans are more frequently, and humans are more frequently, and humans are creating creating creating things faster, organizational things faster, organizational things faster, organizational processes may not processes may not processes may not keep up. The first keep up. The first keep up. The first bottleneck for bottleneck for bottleneck for development environments development environments development environments will be CICD processes, and will be CICD processes, and will be CICD processes, and you may have heard that you may have heard that you may have heard that CPU is becoming a bottleneck CPU is becoming a bottleneck CPU is becoming a bottleneck . Another thing . Another thing is simply the processes is simply the processes is simply the processes that operate in the company. that operate in the company. that operate in the company. How do you trust? How do you How do you trust? How do you How do you trust? How do you check? I think the check? I think the check? I think the previous speaker previous speaker previous speaker talked a lot about talked a lot about talked a lot about trust. Yes, it is the lack of trust. Yes, it is the lack of trust. Yes, it is the lack of trust that trust that trust that slows everything down. And this may slows everything down. And this may slows everything down. And this may be entirely be entirely be entirely justified. Perhaps the justified. Perhaps the justified. Perhaps the AI ​​agent was poorly AI ​​agent was poorly AI ​​agent was poorly designed and designed and designed and shouldn't be trusted.

shouldn't be trusted. shouldn't be trusted. But it works the But it works the But it works the other way around too. What if, other way around too. What if, other way around too. What if, hypothetically, this hypothetically, this hypothetically, this agent did everything agent did everything agent did everything perfectly? But people perfectly? But people perfectly? But people didn't trust him because there didn't trust him because there didn't trust him because there was no proper was no proper was no proper testing, testing, testing, validation, and he didn't validation, and he didn't validation, and he didn't deserve that trust. deserve that trust. deserve that trust. This slows down the process. This slows down the process. And these things have a And these things have a much greater impact. much greater impact. So, again: be So, again: be very careful with very careful with very careful with how you use how you use how you use this tool, this tool, this tool, as you as you as you need to need to need to adjust your actions accordingly. adjust your actions accordingly. So, this again So, this again brings us to the brings us to the brings us to the idea that tokens are idea that tokens are idea that tokens are not a finished product. not a finished product. One of the main One of the main examples I examples I examples I want to highlight—and this want to highlight—and this want to highlight—and this happens more often than happens more often than happens more often than I would like—is I would like—is that that that sometimes spamming with more sometimes spamming with more sometimes spamming with more agents and agents and agents and tokens results tokens results tokens results in the in the in the same task same task same task being executed multiple being executed multiple being executed multiple times. So, one of the times. So, one of the times. So, one of the anti-patterns we've anti-patterns we've anti-patterns we've started to see is that started to see is that while the engineering while the engineering organization generally organization generally organization generally improves improves improves productivity by productivity by productivity by fixing fixing fixing more issues faster and more issues faster and more issues faster and automating their automating their automating their resolution—for example resolution—for example , when a QA team , when a QA team reports a reports a bug and an agent bug and an agent bug and an agent automatically automatically automatically fixes it without fixes it without fixes it without the engineer's involvement—these are great the engineer's involvement—these are great the engineer's involvement—these are great results. But one results. But one results. But one of the common problems of the common problems of the common problems we've started to we've started to we've started to run into is run into is run into is that people that people that people become too become too become too ambitious very quickly. Instead of ambitious very quickly. Instead of ambitious very quickly. Instead of making making making small changes, they small changes, they small changes, they start making start making start making huge rewrites, huge rewrites, huge rewrites, changing half changing half changing half or a quarter of the or a quarter of the or a quarter of the codebase. Sometimes, if it's codebase. Sometimes, if it's codebase. Sometimes, if it's planned and planned and planned and properly properly properly tested, it's tested, it's tested, it's great. But often great. But often great. But often changes occur on

changes occur on changes occur on such a large such a large such a large scale that the bottleneck scale that the bottleneck scale that the bottleneck becomes the becomes the becomes the reviewer who reviewer who reviewer who needs to merge this needs to merge this needs to merge this code. And since everything code. And since everything code. And since everything is being done on such a is being done on such a is being done on such a scale that a person scale that a person scale that a person cannot process it cannot process it , one of the common , one of the common , one of the common anti-patterns that we have anti-patterns that we have anti-patterns that we have begun to observe begun to observe begun to observe is a situation where is a situation where is a situation where someone has already implemented a someone has already implemented a someone has already implemented a solution and the task is solution and the task is solution and the task is supposedly completed. And supposedly completed. And supposedly completed. And because because because so so so much code is generated, issues much code is generated, issues for which code was not for which code was not for which code was not generated generated generated go go go unnoticed and do not unnoticed and do not unnoticed and do not reach the reach the reach the reviewer. So reviewer. So reviewer. So someone needs to someone needs to someone needs to generate the code generate the code generate the code using an agent. And using an agent. And using an agent. And when the agent generates this when the agent generates this when the agent generates this huge pile of code— huge pile of code— which could have been which could have been which could have been much smaller with much smaller with much smaller with proper validation of proper validation of proper validation of prompts and so on— prompts and so on— the reviewer looks the reviewer looks the reviewer looks at it and says, “Hmm, there’s at it and says, “Hmm, there’s at it and says, “Hmm, there’s too much code here. too much code here. too much code here. Can I ask Can I ask Can I ask my agent my agent my agent to highlight the relevant to highlight the relevant to highlight the relevant changes and then changes and then changes and then rework them? And rework them? And rework them? And eventually they eventually they eventually they start start start reworking everything reworking everything reworking everything manually with the manually with the manually with the agent. So they agent. So they agent. So they literally reject the literally reject the literally reject the first first first agent's work. There are agent's work. There are agent's work. There are many good many good many good reasons for this. For example, reasons for this. For example, reasons for this. For example, reviewers want reviewers want reviewers want to maintain control and to maintain control and to maintain control and quality of the core quality of the core quality of the core codebase, but this is a codebase, but this is a codebase, but this is a very common very common very common example of a bottleneck example of a bottleneck example of a bottleneck where, due to a where, due to a where, due to a lack of a lack of a lack of a thoughtful process, thoughtful process, thoughtful process, you just end up adding you just end up adding you just end up adding more and more stuff more and more stuff . And, accordingly, . And, accordingly, . And, accordingly, the speed of work does not the speed of work does not the speed of work does not increase. So, increase. So, increase. So, review your review your review your processes. A good processes. A good processes. A good way to get around this is way to get around this is to implement to implement to implement testing best practices testing best practices testing best practices so you can so you can so you can automate the automate the automate the process and process and process and filter out filter out filter out errors. It's also worth errors. It's also worth errors. It's also worth reminding teams, reminding teams, reminding teams, "Hey, break up your "Hey, break up your "Hey, break up your merges or make them merges or make them merges or make them more modular to make the

more modular to make the review process easier." These are the review process easier." These are the things. These are human things. These are human things. These are human processes. This has nothing processes. This has nothing processes. This has nothing to do with AI. And yes, to do with AI. And yes, to do with AI. And yes, we sometimes get too we sometimes get too we sometimes get too excited about AI. So, excited about AI. So, it's not just a it's not just a code problem. We are starting to code problem. We are starting to code problem. We are starting to see AI agents see AI agents see AI agents being deployed to being deployed to being deployed to create create create presentations. This presentations. This presentations. This presentation was presentation was presentation was entirely entirely entirely AI-generated, from a design perspective. AI-generated, from a design perspective. AI-generated, from a design perspective. I went through I went through I went through several iterations. This several iterations. This several iterations. This was not done in was not done in was not done in one request. I one request. I one request. I went through each went through each went through each page at least 30 page at least 30 page at least 30 times to change times to change times to change the text, but the design the text, but the design the text, but the design was done by the agents. And was done by the agents. And was done by the agents. And sometimes in an organization sometimes in an organization sometimes in an organization you start to you start to you start to notice people who notice people who notice people who just generate just generate just generate entire slides in one entire slides in one entire slides in one request and everything in that request and everything in that request and everything in that vein. And they vein. And they vein. And they start start start spreading it. They spreading it. They spreading it. They start holding start holding start holding meetings. And then the burden meetings. And then the burden meetings. And then the burden falls on everyone else falls on everyone else falls on everyone else to filter out to filter out to filter out where there is where there is where there is inconsistency, where there is inconsistency, where there is inconsistency, where there is distortion of distortion of distortion of information. Why are these information. Why are these information. Why are these two data points two data points two data points completely completely completely different? This is different? This is different? This is another example similar another example similar another example similar to the previous one: people to the previous one: people to the previous one: people generate code, slides, generate code, slides, generate code, slides, or reports, and the burden of or reports, and the burden of or reports, and the burden of verification falls on verification falls on verification falls on the shoulders of another person.

the shoulders of another person. So be very So be very careful about these careful about these careful about these organizational changes. organizational changes. organizational changes. Why am I still Why am I still Why am I still optimistic? I don't think that's optimistic? I don't think that's optimistic? I don't think that's necessarily a necessarily a necessarily a bad thing, because everyone has to bad thing, because everyone has to bad thing, because everyone has to learn. And this is where learn. And this is where learn. And this is where we start to we start to adapt our adapt our processes accordingly. Another processes accordingly. Another processes accordingly. Another important point: there are important point: there are important point: there are many typical many typical many typical examples of examples of examples of the use of AI. You the use of AI. You the use of AI. You know your know your know your subject area better within the subject area better within the subject area better within the organization. And there will be organization. And there will be organization. And there will be cases where AI cases where AI cases where AI can be used can be used can be used to automate a to automate a to automate a process that could process that could process that could save your save your save your organization millions organization millions organization millions or significantly improve or significantly improve or significantly improve productivity, productivity, productivity, something only you, something only you, something only you, not me as an AI expert, know about. not me as an AI expert, know about. I like I like to give to give to give the most famous example of the most famous example of the most famous example of Walmart, or was it Walmart? Walmart, or was it Walmart? Walmart, or was it Walmart? I think it was Walmart. I think it was Walmart. I think it was Walmart. Where they literally Where they literally Where they literally started putting a step stand started putting a step stand started putting a step stand in in in every truck. It every truck. It every truck. It was the simplest was the simplest was the simplest change, but it change, but it change, but it saved them saved them saved them millions in costs millions in costs millions in costs because it reduced because it reduced because it reduced the number of the number of the number of back injuries when handling back injuries when handling back injuries when handling heavy loads. It's heavy loads. It's heavy loads. It's something like this something like this , but using , but using , but using AI. And it may AI. And it may AI. And it may not even be the best not even be the best not even be the best AI model, but it AI model, but it AI model, but it could have the biggest could have the biggest could have the biggest impact. Yes. Yes. So, impact. Yes. Yes. So, impact. Yes. Yes. So, building on my building on my building on my previous thought, one previous thought, one previous thought, one of the terms that has of the terms that has of the terms that has become popular is " become popular is " slop grenade." This is slop grenade." This is slop grenade." This is when you generate a when you generate a when you generate a bunch of stuff bunch of stuff bunch of stuff using AI and then

using AI and then using AI and then translate it to translate it to translate it to someone else. Don't someone else. Don't someone else. Don't let them become let them become let them become "slop grenades." It's a "slop grenades." It's a "slop grenades." It's a matter of culture and matter of culture and matter of culture and processes. Of course. So, processes. Of course. So, processes. Of course. So, there are a few basic there are a few basic there are a few basic rules that I would rules that I would rules that I would implement, although it's implement, although it's implement, although it's quite subjective. quite subjective. quite subjective. People will disagree People will disagree . To prevent “slip . To prevent “slip grenades,” demand grenades,” demand grenades,” demand that people be held accountable that people be held accountable that people be held accountable for what they for what they for what they create or create or create or share. share. share. For example, For example, For example, I am personally responsible for these slides I am personally responsible for these slides I am personally responsible for these slides , even for the , even for the , even for the generated text. generated text. generated text. If there is a problem, it is If there is a problem, it is If there is a problem, it is my fault. It's not my my fault. It's not my my fault. It's not my AI agent that AI agent that AI agent that has a problem. I has a problem. I has a problem. I can't just can't just can't just pass it on to the pass it on to the pass it on to the agent. I must take agent. I must take agent. I must take responsibility. The responsibility. The responsibility. The same goes for code same goes for code same goes for code and anything that was and anything that was and anything that was generated. You also generated. You also generated. You also need to measure need to measure need to measure results, not results, not results, not usage levels. usage levels. It's more complicated, and I don't It's more complicated, and I don't want to go want to go want to go into detail, but there is a simple into detail, but there is a simple into detail, but there is a simple formula that you can formula that you can formula that you can take as a rule: take as a rule: take as a rule: the number of tokens per the number of tokens per the number of tokens per price divided by the price divided by the price divided by the actual result. actual result.

This helps This helps to determine many to determine many to determine many things. Smaller or more things. Smaller or more things. Smaller or more efficient models, efficient models, efficient models, faster or more powerful faster or more powerful faster or more powerful models, etc. But what models, etc. But what models, etc. But what I don't like about I don't like about I don't like about this approach is that I'm this approach is that I'm this approach is that I'm comparing AI to comparing AI to comparing AI to supplements or supplements or supplements or steroids that steroids that steroids that enhance something. Imagine, enhance something. Imagine, enhance something. Imagine, for example, for example, for example, construction. You can construction. You can construction. You can use this use this use this to increase the to increase the to increase the muscle mass of muscle mass of muscle mass of workers. workers. workers. For example, although For example, although For example, although it is ethically questionable. I it is ethically questionable. I it is ethically questionable. I understand that this is not true, understand that this is not true, understand that this is not true, but just as an example but just as an example but just as an example for thought. Some for thought. Some for thought. Some teams teams teams use this and use this and use this and improve improve improve productivity, others don't productivity, others don't . But even without . But even without . But even without that, it's obvious that that, it's obvious that that, it's obvious that some teams are better than some teams are better than some teams are better than others. So AI is here as others. So AI is here as others. So AI is here as an additive, it doesn't an additive, it doesn't an additive, it doesn't necessarily change necessarily change necessarily change everything radically. You everything radically. You everything radically. You should remember that should remember that should remember that results are difficult results are difficult results are difficult to measure, but they are something to measure, but they are something to measure, but they are something that you can start to that you can start to that you can start to observe and observe and observe and evaluate. evaluate. The most important thing— going back to the going back to the going back to the first step: first step: first step: measure. You know the measure. You know the measure. You know the token spending per token spending per token spending per department, per team, and you department, per team, and you department, per team, and you can see who is can see who is can see who is performing better based performing better based performing better based on on on token consumption. You will be able token consumption. You will be able token consumption. You will be able to see exactly what to see exactly what to see exactly what works. Yes. As I works. Yes. As I works. Yes. As I said, this is an said, this is an said, this is an imperfect method imperfect method imperfect method of measurement. So yes.

of measurement. So yes. of measurement. So yes. Here is a general Here is a general Here is a general overview of the situation. And overview of the situation. And overview of the situation. And if you want to if you want to if you want to start this process start this process start this process within your within your within your organization, you organization, you organization, you can reach out to can reach out to can reach out to me or my team, me or my team, me or my team, or here are six questions to get you or here are six questions to get you or here are six questions to get you started: started: started: which team is spending how much which team is spending how much which team is spending how much , what needs to be , what needs to be , what needs to be tested to tested to tested to use the use the use the cheaper model, what is cheaper model, what is cheaper model, what is your baseline your baseline your baseline workload, what is the workload, what is the workload, what is the default model default model default model for for for each team, and for each team, and for each team, and for which workloads are the which workloads are the user waiting and user waiting and you need something you need something you need something faster. And what part faster. And what part faster. And what part of what we of what we of what we generate is useful? generate is useful? These are kind of These are kind of These are kind of guiding questions guiding questions guiding questions to start to start to start improving your improving your improving your tokenomics. Yes. So, tokenomics. Yes. So, tokenomics. Yes. So, questions and answers. questions and answers. Yes. When you mentioned When you mentioned fine-tuning fine-tuning fine-tuning models, it's... models, it's... models, it's... Yes, I'm trying. Is Yes, I'm trying. Is Yes, I'm trying. Is this SFT-type training this SFT-type training this SFT-type training or are there other methods, or are there other methods, or are there other methods, depending on the depending on the depending on the specific specific specific use case or use case or use case or available data? available data? Okay, generally speaking, Okay, generally speaking, it's usually SFT, but I it's usually SFT, but I it's usually SFT, but I don't think...um...We don't think...um...We don't think...um...We do provide do provide do provide pre-training services, pre-training services, pre-training services, and most often it's SFT.

and most often it's SFT. There are many There are many different methods now. I different methods now. I different methods now. I know that RL ( know that RL ( know that RL ( reinforcement learning) is reinforcement learning) is reinforcement learning) is very popular right now, very popular right now, very popular right now, especially in especially in especially in programming, because it's programming, because it's programming, because it's easy to generate easy to generate easy to generate data in a loop. But we data in a loop. But we data in a loop. But we understand that for understand that for understand that for many organizations many organizations , for example, in New , for example, in New York, in financial York, in financial York, in financial institutions, etc., there institutions, etc., there institutions, etc., there is no real RL is no real RL environment for environment for environment for financial tasks. financial tasks. Therefore, SFT is more commonly Therefore, SFT is more commonly used. used. There are also other There are also other organizations like organizations like organizations like Adaptation AI. They Adaptation AI. They Adaptation AI. They are starting are starting are starting to automate the to automate the to automate the refinement of refinement of refinement of datasets for training datasets for training datasets for training models, whether models, whether models, whether based on SFT or with based on SFT or with machine learning in mind. Maybe this isn't quite the Maybe this isn't quite the right question. right question. right question. In general, what In general, what In general, what people should people should people should focus on is focus on is focus on is evaluations (Evals). Here is a evaluations (Evals). Here is a evaluations (Evals). Here is a good example. When the good example. When the good example. When the Jeff model was released, we Jeff model was released, we Jeff model was released, we made an open-source project made an open-source project made an open-source project called Simple Jeff called Simple Jeff called Simple Jeff that replicates the Jeff API for that replicates the Jeff API for that replicates the Jeff API for existing existing existing open-source models.

open-source models. open-source models. Many quickly Many quickly Many quickly realized that the Qwen 27B realized that the Qwen 27B copes very well with this copes very well with this task. And several task. And several task. And several teams began teams began teams began to train Qwen 27B. We at to train Qwen 27B. We at to train Qwen 27B. We at our company our company our company try to approach this this try to approach this this try to approach this this way: we way: we way: we want to take want to take any model and just any model and just any model and just build an API on top of build an API on top of build an API on top of it. We started with the it. We started with the it. We started with the approach of, "Hey, we approach of, "Hey, we approach of, "Hey, we see benchmarks, we see benchmarks, we see benchmarks, we see the existing see the existing see the existing model." Why model." Why model." Why don't we don't we don't we apply apply prompt settings instead? This is prompt settings instead? This is n't even a refresher course. And n't even a refresher course. And n't even a refresher course. And then, two then, two then, two weeks later, when Hugging Face weeks later, when Hugging Face weeks later, when Hugging Face ran all the ran all the ran all the benchmarks at scale, benchmarks at scale, benchmarks at scale, like over 10,000 like over 10,000 like over 10,000 questions, etc., questions, etc., questions, etc., somehow my Qwen 27B model somehow my Qwen 27B model somehow my Qwen 27B model with a customized with a customized with a customized prompt beat prompt beat prompt beat most of the pre-trained most of the pre-trained most of the pre-trained 27B models. So yes, this is 27B models. So yes, this is 27B models. So yes, this is not bragging, because I am not bragging, because I am not bragging, because I am confident that confident that confident that my model is now my model is now my model is now outperformed by the new outperformed by the new outperformed by the new pre-trained model from pre-trained model from pre-trained model from Cloudflare, built in a more Cloudflare, built in a more Cloudflare, built in a more correct way. But correct way. But correct way. But the point is rather the point is rather the point is rather that the that the that the setup method may setup method may setup method may not be the main not be the main not be the main issue. issue. issue. The question is more like: "What are The question is more like: "What are The question is more like: "What are your grades?" And what your grades?" And what your grades?" And what results did they results did they results did they lead to? lead to? lead to? And following that: do And following that: do And following that: do you need to collect you need to collect you need to collect “golden” tracks for the “golden” tracks for the “golden” tracks for the dataset if you’re dataset if you’re dataset if you’re doing SFT or something doing SFT or something doing SFT or something like that, or can you like that, or can you like that, or can you just…let’s say, like in just…let’s say, like in just…let’s say, like in your previous your previous your previous task, the task, the task, the frontier models are frontier models are frontier models are about 80% accurate, about 80% accurate, about 80% accurate, right? So, maybe 20 right? So, maybe 20 % of the time they % of the time they % of the time they make make make mistakes. If you mistakes. If you mistakes. If you retrain on retrain on retrain on this, wo this, wo this, wo n't the retrained model also be n't the retrained model also be n't the retrained model also be wrong 20% ​​of the wrong 20% ​​of the time? Should time? Should time? Should you somehow you somehow you somehow filter this out and filter this out and filter this out and leave leave leave only the " only the " golden" trails for training, or golden" trails for training, or golden" trails for training, or can you just can you just can you just throw all the trails away and throw all the trails away and throw all the trails away and train?

train? This is more of my This is more of my personal opinion on personal opinion on personal opinion on the technique, because usually the technique, because usually the technique, because usually I try to I try to I try to split the split the split the dataset because people are dataset because people are dataset because people are lazy in terms of lazy in terms of lazy in terms of filtering. So I'm filtering. So I'm filtering. So I'm trying to at least trying to at least trying to at least get a set of " get a set of " gold" tracks. And this gold" tracks. And this gold" tracks. And this set can be set can be set can be much smaller, and we much smaller, and we much smaller, and we just retrain just retrain just retrain it with much it with much it with much more weight. But there more weight. But there more weight. But there have been many articles, have been many articles, have been many articles, especially on “ especially on “ thinking tokens,” thinking tokens,” thinking tokens,” that suggest that that suggest that that suggest that even failed even failed even failed tasks are useful. tasks are useful. Even a smaller model benefits from such benefits from such training when retraining, as training when retraining, as training when retraining, as it improves the it improves the it improves the thinking process, even if thinking process, even if thinking process, even if the answer received the answer received the answer received was incorrect. This is a was incorrect. This is a was incorrect. This is a great example of great example of great example of trying and trying and trying and measuring, and it measuring, and it measuring, and it works 70% of the time. works 70% of the time. works 70% of the time. Sometimes you need Sometimes you need Sometimes you need to increase the proportion of " to increase the proportion of " golden" routes, but this is a golden" routes, but this is a golden" routes, but this is a general rule. general rule.

general rule. I think for many I think for many I think for many customers who customers who customers who are trying are trying are trying to SFT a smaller to SFT a smaller to SFT a smaller model, the main model, the main model, the main rule they rule they rule they try try try to follow is, " to follow is, " to follow is, " Can I maintain Can I maintain Can I maintain roughly the same roughly the same roughly the same level of level of level of performance?" And performance?" And performance?" And maybe the score will drop maybe the score will drop maybe the score will drop by 1%, or even by 1%, or even improve by 10% in some cases, improve by 10% in some cases, because the base model because the base model because the base model was not was not was not specialized for specialized for specialized for these tasks. And more these tasks. And more these tasks. And more importantly, does importantly, does importantly, does this reduce my bill by 100 this reduce my bill by 100 this reduce my bill by 100 times? So times? So times? So this is usually the appropriate this is usually the appropriate this is usually the appropriate motivation. Yes, motivation. Yes, motivation. Yes, reference samples are reference samples are reference samples are critical critical critical if you want to go if you want to go if you want to go beyond the basic beyond the basic beyond the basic level. And sometimes that level. And sometimes that level. And sometimes that requires people requires people requires people to look at the to look at the to look at the data themselves, and as much as I would data themselves, and as much as I would data themselves, and as much as I would like clients to like clients to like clients to look at the data more often look at the data more often look at the data more often , people don't , people don't , people don't like that. like that. Greetings. How would you Greetings. How would you deal with the deal with the deal with the code overhead that is code overhead that is code overhead that is mostly caused mostly caused mostly caused by Claude Code and all the by Claude Code and all the by Claude Code and all the integrations within it? integrations within it? Because then your model is Because then your model is completely tied completely tied completely tied to the Claude ecosystem, I to the Claude ecosystem, I to the Claude ecosystem, I think. Do you have any think. Do you have any think. Do you have any advice on how to act in advice on how to act in advice on how to act in such situations?

such situations? Yes, unfortunately, Claude Code is an example of full is an example of full is an example of full binding. So sometimes the binding. So sometimes the binding. So sometimes the first step is first step is first step is actually convincing actually convincing actually convincing them to change their them to change their them to change their toolkit. toolkit. It seems that It seems that the toolkit has the toolkit has the toolkit has become much more become much more become much more flexible now. We have seen flexible now. We have seen flexible now. We have seen organizations that are organizations that are organizations that are completely locked completely locked completely locked into one system. into one system. into one system. Like "I'll Like "I'll only use Devin or only only use Devin or only M." We have seen M." We have seen M." We have seen organizations that are organizations that are organizations that are more flexible. They more flexible. They more flexible. They literally run literally run literally run their own agent their own agent their own agent systems. And my systems. And my systems. And my reaction: “Why?” “ reaction: “Why?” “ Why are you building Why are you building Why are you building your own agent your own agent your own agent system from scratch?” But, system from scratch?” But, system from scratch?” But, for example, a for example, a for example, a popular option that popular option that people are switching to is Pi. And that's people are switching to is Pi. And that's also because there's a also because there's a also because there's a lot of lot of lot of noise around it. People have shown noise around it. People have shown that even with the same that even with the same that even with the same model, the model, the model, the token costs are much lower token costs are much lower token costs are much lower due to simplicity and due to simplicity and due to simplicity and efficiency. And I think efficiency. And I think efficiency. And I think the beauty of the the beauty of the the beauty of the Pi agent system for Pi agent system for Pi agent system for programming is that programming is that programming is that it's easy it's easy it's easy to adapt because it's to adapt because it's to adapt because it's so lightweight. We've seen so lightweight. We've seen so lightweight. We've seen many teams many teams many teams simply start simply start simply start prompting their prompting their prompting their agents to make agents to make agents to make changes. And once they changes. And once they changes. And once they have that, they have that, they have that, they begin a cycle begin a cycle begin a cycle of experimentation and of experimentation and of experimentation and change. A common change. A common change. A common way to incentivize way to incentivize way to incentivize them to change is them to change is them to change is to present a pretty to present a pretty to present a pretty good model that is good model that is good model that is very fast, very fast, very fast, like 100 tokens like 100 tokens like 100 tokens per second. And suddenly per second. And suddenly per second. And suddenly they start they start they start trying it. Some trying it. Some trying it. Some start by start by start by still using Claude still using Claude still using Claude Code as a planner. And Code as a planner. And Code as a planner. And then they instruct then they instruct then they instruct Claude Code Claude Code Claude Code to use to use to use this Pi agent with this

this Pi agent with this this Pi agent with this model for model for model for implementation. Hence implementation. Hence implementation. Hence the separation the separation the separation of planner and of planner and of planner and implementation. That's right, that's where the implementation. That's right, that's where the transition begins to happen. transition begins to happen. Changing Changing Changing people's agents is not easy, but it is people's agents is not easy, but it is people's agents is not easy, but it is possible. possible. Um, yeah, an example of Um, yeah, an example of that would be that would be that would be maybe exposing maybe exposing maybe exposing this smaller agent this smaller agent this smaller agent as an MCP inside the cloud as an MCP inside the cloud as an MCP inside the cloud or having these or having these or having these tools where you tools where you tools where you can offload... can offload... I guess would I guess would I guess would you say that you say that you say that could work? It's could work? It's more complicated with MCP more complicated with MCP because MCP agents, if because MCP agents, if because MCP agents, if not configured not configured not configured correctly, don't have correctly, don't have correctly, don't have access to your access to your access to your file system. So file system. So file system. So usually it's something usually it's something usually it's something simpler like simpler like simpler like python installed and python installed and python installed and they run a they run a they run a batch file batch file batch file like "python this.py". like "python this.py". like "python this.py". Yes, that's usually the case. Yes, that's usually the case. Yes, that's usually the case. Thank you. Thank you. Yes. Can you Can you share, if share, if share, if possible, some possible, some possible, some case case case studies on studies on studies on caching solutions caching solutions caching solutions where the approach was very where the approach was very where the approach was very deliberate and the deliberate and the deliberate and the results that were results that were results that were obtained for obtained for obtained for specific customers, specific customers, specific customers, even if it's even if it's even if it's anonymous?

anonymous? What problem are What problem are you talking about? You you talking about? You you talking about? You do one do one do one caching...Okay, caching...Okay, caching...Okay, your question is your question is your question is more about what you more about what you more about what you want to anonymize want to anonymize ...Wait, you don't ...Wait, you don't ...Wait, you don't want the want the customer's sensitive information customer's sensitive information to be cached. to be cached. No, I mean, with clients No, I mean, with clients you've worked with, you've worked with, you've worked with, do you have any case studies do you have any case studies you can you can you can share about share about share about the results the results the results they've been able to achieve they've been able to achieve they've been able to achieve with a thoughtful with a thoughtful with a thoughtful approach to caching approach to caching approach to caching and different and different and different caching methods? caching methods? I think we can I think we can discuss this separately, discuss this separately, discuss this separately, but in general, for our but in general, for our but in general, for our large customers, we large customers, we large customers, we implement solutions implement solutions implement solutions with a dedicated cache. This with a dedicated cache. This with a dedicated cache. This solves many solves many privacy issues privacy issues because the cache is because the cache is because the cache is only for a only for a only for a specific client. specific client. specific client. Yes, and of course there is Yes, and of course there is prompt formatting, but I prompt formatting, but I might need to might need to might need to dive deeper into dive deeper into dive deeper into what exactly you what exactly you what exactly you are trying to ask are trying to ask . The short answer is . The short answer is yes, but it can be yes, but it can be yes, but it can be interpreted in interpreted in different ways. Yes. different ways. Yes. different ways. Yes. Steeply. Sorry about that. Steeply. Sorry about that. What are your thoughts on What are your thoughts on turn-based turn-based turn-based or task- or task- oriented oriented oriented automatic automatic automatic routing?

routing? Oh, I generally Oh, I generally lean more towards task- lean more towards task- oriented. oriented. Mainly because it is Mainly because it is much easier much easier much easier to control to control to control expectations at the expectations at the expectations at the department or department or department or organization level. This organization level. This organization level. This brings us back to the fact brings us back to the fact brings us back to the fact that, for example, a that, for example, a that, for example, a department might department might department might have its own model for have its own model for have its own model for labeling, and I labeling, and I labeling, and I automatically automatically automatically route requests to route requests to route requests to that model for them. I that model for them. I that model for them. I know there's a lot of know there's a lot of know there's a lot of buzz around buzz around buzz around step-by-step step-by-step step-by-step automatic automatic automatic routing right now. One of the routing right now. One of the routing right now. One of the problems with step-by-step problems with step-by-step problems with step-by-step routing routing routing is that is that is that you need to limit you need to limit you need to limit the number of models to, the number of models to, the number of models to, say, say, say, two or three at most. If not, two or three at most. If not, two or three at most. If not, your cache will be extremely your cache will be extremely your cache will be extremely inefficient to inefficient to inefficient to use. use. use. For example, we saw For example, we saw For example, we saw step-by-step step-by-step step-by-step routing for GRM routing for GRM routing for GRM 5.3 and GRM 5.3 flash. This is 5.3 and GRM 5.3 flash. This is 5.3 and GRM 5.3 flash. This is logical. But if you logical. But if you logical. But if you add 30 models and add 30 models and add 30 models and they constantly they constantly they constantly change at change at change at every step, the every step, the every step, the low low low cache hit rate will start to cache hit rate will start to cache hit rate will start to destroy your destroy your destroy your optimization. optimization. optimization. Yes, I've seen this Yes, I've seen this Yes, I've seen this before. I saw it at the before. I saw it at the before. I saw it at the query level and it query level and it query level and it just didn't make sense just didn't make sense just didn't make sense and then and then and then Yes, but two models are Yes, but two models are Yes, but two models are usually enough. Er usually enough. Er , tasks and , tasks and , tasks and organization make organization make organization make the most sense. Yes.

the most sense. Yes. the most sense. Yes. Yes. Yes, I agree. Yes. Yes, I agree. OK. Okay. OK. Okay. Thank you very much, and if anyone Thank you very much, and if anyone Thank you very much, and if anyone wants to find me to wants to find me to wants to find me to ask additional ask additional ask additional questions, come over questions, come over questions, come over here or find here or find here or find me on the side once the me on the side once the me on the side once the next speaker is on next speaker is on next speaker is on . Yes. . Yes. . Yes. Thank you very much.