Same Model, Different Speed: Why Your Inference Provider Matters — FriendliAI

Well, let's begin. So, Well, let's begin. So, congratulations to everyone. My congratulations to everyone. My congratulations to everyone. My name is Yunmo. I am a name is Yunmo. I am a founding engineer at Friendly AI. founding engineer at Friendly AI. founding engineer at Friendly AI. Together with our Together with our Together with our team, I am building a team, I am building a team, I am building a logical logical logical inference engine to inference engine to inference engine to optimize optimize optimize performance. Today I'll performance. Today I'll performance. Today I'll talk about how we talk about how we talk about how we built a fast and built a fast and built a fast and reliable API for reliable API for reliable API for advanced advanced advanced open-weight models. Oh, open-weight models. Oh, open-weight models. Oh, yes. Thank you. Okay, yes. Thank you. Okay, yes. Thank you. Okay, before we before we before we get too deep, let's take a get too deep, let's take a get too deep, let's take a quick show of quick show of quick show of hands vote. So, hands vote. So, hands vote. So, who here uses who here uses who here uses closed-source advanced closed-source advanced closed-source advanced models like GPT or models like GPT or models like GPT or Claude? Good. Most Claude? Good. Most Claude? Good. Most of you are already of you are already of you are already using them. And using them. And using them. And who here mainly who here mainly who here mainly uses models uses models uses models with open scales? with open scales? Yes. Yes. Less than Yes. Yes. Less than closed models. closed models. Personally, I've Personally, I've been using a been using a been using a mix of both lately, but I'm mix of both lately, but I'm mix of both lately, but I'm gradually shifting gradually shifting gradually shifting more more more workloads to workloads to workloads to open-scale models. open-scale models. The reason is that The reason is that open-scale models open-scale models have recently crossed a have recently crossed a certain threshold.

certain threshold. certain threshold. For example, in tests For example, in tests For example, in tests of coding, logical of coding, logical of coding, logical reasoning, and reasoning, and reasoning, and agentic capabilities, agentic capabilities, agentic capabilities, models like GLM 5.2 models like GLM 5.2 models like GLM 5.2 compete on compete on compete on par with leading par with leading par with leading proprietary proprietary proprietary models. As you models. As you models. As you can see on this can see on this can see on this Code Arena leaderboard, GLM 5.2 Code Arena leaderboard, GLM 5.2 Code Arena leaderboard, GLM 5.2 ranks even higher ranks even higher ranks even higher than Claude Opus 4.8. than Claude Opus 4.8. It is It is right below Claude Fable 5. right below Claude Fable 5. Similarly, in Similarly, in various various various coding, coding, coding, terminal, and terminal, and tool usage tests, GLM 5.2 and tool usage tests, GLM 5.2 and Minimax M3 show Minimax M3 show Minimax M3 show strong results. strong results. strong results. So the question is no longer just So the question is no longer just So the question is no longer just whether whether open- open- scale models are good enough. For many scale models are good enough. For many scale models are good enough. For many work tasks. They are work tasks. They are work tasks. They are already like that. So the already like that. So the already like that. So the next question is next question is how quickly and how quickly and how quickly and reliably we can reliably we can reliably we can launch them into launch them into launch them into production. Benchmark numbers production. Benchmark numbers alone do not provide a clear alone do not provide a clear understanding of how understanding of how understanding of how things actually work. things actually work. So I conducted a simple So I conducted a simple practical practical practical experiment to experiment to experiment to see how these see how these see how these models work in models work in models work in practice. I asked practice. I asked practice. I asked both GLM 5.2 and Opus 4.8 both GLM 5.2 and Opus 4.8 both GLM 5.2 and Opus 4.8 to create a to create a tower defense game like this. You tower defense game like this. You tower defense game like this. You can see that both can see that both can see that both models produced a models produced a models produced a working implementation with working implementation with working implementation with similar structure and similar structure and similar structure and quality. Both have a map quality. Both have a map quality. Both have a map and turrets that shoot and turrets that shoot and turrets that shoot at enemies within at enemies within at enemies within range. Looks range. Looks range. Looks pretty great pretty great pretty great considering I considering I considering I did it in one did it in one did it in one query. But when we query. But when we query. But when we look at the cost, the look at the cost, the look at the cost, the difference becomes very difference becomes very difference becomes very obvious. GLM 5.2

obvious. GLM 5.2 obvious. GLM 5.2 turned out to be more than turned out to be more than turned out to be more than 5.6 times cheaper for 5.6 times cheaper for 5.6 times cheaper for this task. And in this task. And in this task. And in agent-based agent-based agent-based workflows, where workflows, where workflows, where multiple multiple multiple iterations are required, this gap iterations are required, this gap iterations are required, this gap grows rapidly. So, it's grows rapidly. So, it's grows rapidly. So, it's not just about the not just about the not just about the quality of the model. The point is that quality of the model. The point is that quality of the model. The point is that you can you can you can afford to afford to afford to launch on a large launch on a large launch on a large scale. After scale. After scale. After the quality of the model and the quality of the model and the quality of the model and the cost, the next the cost, the next the cost, the next issue is the quality of issue is the quality of issue is the quality of the supplier. And the supplier. And the supplier. And supplier quality supplier quality is actually is actually is actually productivity. So, productivity. So, productivity. So, even if the even if the even if the model weights are the same, model weights are the same, model weights are the same, latency, latency, latency, throughput, and throughput, and throughput, and reliability can reliability can reliability can vary significantly vary significantly vary significantly depending on where depending on where depending on where you run it. Here is you run it. Here is you run it. Here is a screenshot from OpenRouter that a screenshot from OpenRouter that a screenshot from OpenRouter that shows that these shows that these shows that these differences can be differences can be differences can be measured. So, if you measured. So, if you measured. So, if you are building an are building an are building an AI product, AI product, AI product, you should you should you should ask not only ask not only ask not only which model which model which model to use, but also to use, but also to use, but also which provider which provider which provider provides the best provides the best provides the best performance, performance, performance, reliability, and price. reliability, and price. reliability, and price. So, I'll show you another So, I'll show you another So, I'll show you another example. Here I example. Here I example. Here I use the same use the same use the same model, GLM 5.2, and the model, GLM 5.2, and the model, GLM 5.2, and the same same same coding agent workflow.

coding agent workflow. coding agent workflow. The challenge is to The challenge is to The challenge is to create a create a create a simple simple simple Candy Crush-style mobile game, which we Candy Crush-style mobile game, which we Candy Crush-style mobile game, which we call Moto Crush. When call Moto Crush. When call Moto Crush. When you look at this, on you look at this, on you look at this, on the left—Friendly AI. It the left—Friendly AI. It the left—Friendly AI. It generates a game in less than generates a game in less than generates a game in less than 2 minutes. Another 2 minutes. Another 2 minutes. Another provider takes provider takes provider takes more than 3 minutes. So, more than 3 minutes. So, more than 3 minutes. So, the model is the same, the model is the same, the model is the same, we have the same agent, but the we have the same agent, but the we have the same agent, but the provider was different. provider was different. Modeler tools Modeler tools and coding agent and coding agent are exactly what are exactly what users experience. In this users experience. In this simple example, we simple example, we simple example, we saw that even with the saw that even with the saw that even with the same model and same model and same model and task, the experience task, the experience task, the experience can can can vary significantly depending vary significantly depending vary significantly depending on the provider. on the provider. on the provider. The difference is due to the The difference is due to the The difference is due to the entire entire entire inference stack. The inference stack. The inference stack. The model architecture, model architecture, model architecture, caching system, caching system, routing policy, routing policy, scheduler, scheduler, quantization strategy, GPU cores— all of this together forms what all of this together forms what the developer actually gets. This is the the developer actually gets. This is the essence of the value of essence of the value of essence of the value of Friendly AI. We don't just Friendly AI. We don't just Friendly AI. We don't just provide a model API.

provide a model API. provide a model API. Friendly AI is built as a Friendly AI is built as a Friendly AI is built as a full-stack full-stack full-stack inference engine for inference engine for inference engine for generative AI generative AI models. This is important models. This is important models. This is important because fast and because fast and because fast and reliable output reliable output is not just about is not just about is not just about optimization. It's optimization. It's optimization. It's batch processing, batch processing, batch processing, scheduling, scheduling, scheduling, quantization, quantization, quantization, speculative speculative speculative coding, and kernels—it coding, and kernels—it all has to work all has to work all has to work together. Let me together. Let me together. Let me briefly explain how briefly explain how briefly explain how we built this and how we we built this and how we we built this and how we achieve advanced achieve advanced achieve advanced levels of levels of levels of inference performance. In modern inference performance. In modern inference performance. In modern agent applications, agent applications, agent applications, input data is often input data is often input data is often very long. They very long. They very long. They easily exceed 10 easily exceed 10 easily exceed 10 thousand tokens, thousand tokens, thousand tokens, while the original data is while the original data is while the original data is relatively short. And, relatively short. And, relatively short. And, more importantly, a significant more importantly, a significant more importantly, a significant portion of the input portion of the input portion of the input data is repeated across data is repeated across data is repeated across different queries. In different queries. In different queries. In many real-world many real-world workloads, from 50% to 80-90% of tokens are to 80-90% of tokens are to 80-90% of tokens are identical across identical across identical across requests. From a systems requests. From a systems requests. From a systems perspective, this means perspective, this means that most of the that most of the that most of the computational computational computational cost falls on the cost falls on the cost falls on the prefill stage, when prefill stage, when prefill stage, when the model processes all the the model processes all the the model processes all the input data and builds the input data and builds the input data and builds the KV cache. So, if we KV cache. So, if we recalculate this recalculate this shared prefix every time, we shared prefix every time, we shared prefix every time, we waste both waste both waste both waiting time and waiting time and waiting time and GPU resources. So, when a new GPU resources. So, when a new GPU resources. So, when a new query shares a query shares a query shares a prefix with a prefix with a prefix with a previous one, we can previous one, we can previous one, we can reuse reuse reuse these tensors, these tensors, these tensors, skip the skip the skip the corresponding corresponding corresponding prefix calculation, prefix calculation, prefix calculation, directly directly directly reduce the TTFT, and reduce the TTFT, and reduce the TTFT, and avoid recalculating a avoid recalculating a avoid recalculating a large portion of the large portion of the large portion of the input data. However,

input data. However, input data. However, for this to work at for this to work at for this to work at scale, scale, scale, two key two key two key mechanisms are needed. The first is mechanisms are needed. The first is cache-aware routing. In a cache-aware routing. In a distributed system, the distributed system, the distributed system, the KV cache is not KV cache is not KV cache is not centralized. It centralized. It centralized. It is stored on is stored on is stored on separate replicas. separate replicas. separate replicas. Therefore, the router Therefore, the router Therefore, the router should direct should direct should direct incoming requests to the incoming requests to the incoming requests to the replica that already replica that already replica that already contains the corresponding contains the corresponding contains the corresponding prefix in its cache prefix in its cache . Otherwise, we will lose . Otherwise, we will lose . Otherwise, we will lose the advantage of the advantage of reuse. The second is the cache hierarchy. KV is the cache hierarchy. KV cache is extremely cache is extremely cache is extremely memory intensive memory intensive . For large models . For large models . For large models and long contexts, and long contexts, and long contexts, the size of the KV cache may the size of the KV cache may the size of the KV cache may exceed the size of the exceed the size of the exceed the size of the model weights. As you know, model weights. As you know, model weights. As you know, GPU video memory is GPU video memory is GPU video memory is the fastest, but it is the fastest, but it is the fastest, but it is also a very limited also a very limited also a very limited resource. We resource. We resource. We view cache as a view cache as a view cache as a hierarchy: hot cache hierarchy: hot cache hierarchy: hot cache remains in remains in remains in GPU memory, warm cache remains in GPU memory, warm cache remains in GPU memory, warm cache remains in CPU RAM, and CPU RAM, and CPU RAM, and colder cache can be colder cache can be colder cache can be offloaded to NVMe offloaded to NVMe offloaded to NVMe or disk space. or disk space. The system dynamically The system dynamically offloads less- offloads less- offloads less- used used used blocks and blocks and blocks and preloads them when preloads them when preloads them when they become they become they become needed.

needed. needed. By combining By combining cache-aware routing and cache-aware routing and hierarchical hierarchical hierarchical memory management, we significantly memory management, we significantly memory management, we significantly reduce the reduce the reduce the prefix cost, lower TTFT, prefix cost, lower TTFT, prefix cost, lower TTFT, and make and make and make long context processing long context processing long context processing much more efficient. much more efficient. Next is Next is interactive interactive interactive speed. speed. Standard Standard autoregressive autoregressive autoregressive decoding generates decoding generates decoding generates one token per pass, one token per pass, one token per pass, making making making decoding decoding decoding sequential, which sequential, which often causes the GPU to be underutilized during this process . Speculative . Speculative decoding changes decoding changes decoding changes the nature of this problem. the nature of this problem. Instead of Instead of generating one generating one generating one token and stopping, token and stopping, token and stopping, it proposes several it proposes several it proposes several candidates up front, the candidates up front, the candidates up front, the target model target model target model checks them in one checks them in one checks them in one pass, and if the tokens pass, and if the tokens pass, and if the tokens are accepted, they are are accepted, they are are accepted, they are given away given away given away for free. This for free. This for free. This improves improves improves generation time by one generation time by one generation time by one output token without output token without output token without changing the quality of the changing the quality of the changing the quality of the model output. There are different model output. There are different model output. There are different ways to create ways to create ways to create candidates. One candidates. One candidates. One way is through way is through way is through model-based speculation.

model-based speculation. It uses a It uses a small auxiliary small auxiliary small auxiliary model to model to model to predict predict predict future tokens; future tokens; future tokens; the advantage is that it the advantage is that it the advantage is that it gives a higher gives a higher gives a higher acceptance rate, but the price acceptance rate, but the price acceptance rate, but the price is additional is additional is additional computation. Another computation. Another computation. Another method is speculation method is speculation method is speculation without a model. It without a model. It without a model. It uses templates uses templates uses templates from the prompt from the prompt from the prompt or previous or previous or previous output. It has output. It has output. It has less less less overhead, but overhead, but overhead, but works best only when works best only when works best only when the output resembles the the output resembles the the output resembles the existing context. An existing context. An important aspect is important aspect is compromise. compromise. compromise. Speculation can Speculation can Speculation can improve latency improve latency improve latency per request, but per request, but per request, but validation adds validation adds validation adds overhead to the overhead to the overhead to the packet. Therefore, the best packet. Therefore, the best packet. Therefore, the best strategy is not strategy is not strategy is not static. At Friendly AI, we static. At Friendly AI, we static. At Friendly AI, we take a take a take a hybrid approach. hybrid approach. hybrid approach. The runtime The runtime The runtime can dynamically can dynamically can dynamically choose between choose between model-based and non- model-based and non- model-based speculation, depending on the model-based speculation, depending on the model-based speculation, depending on the load and load and load and query characteristics query characteristics . So, we also . So, we also . So, we also see see see speculation as something that speculation as something that speculation as something that can be improved over can be improved over can be improved over time. So, thanks to time. So, thanks to time. So, thanks to self-learning self-learning self-learning speculation, the draft speculation, the draft speculation, the draft model can model can model can adapt to adapt to adapt to real real real service traffic. So, service traffic. So, service traffic. So, the speculator is constantly the speculator is constantly the speculator is constantly refining their refining their refining their predictions about the predictions about the workload that workload that actually runs in actually runs in actually runs in your production.

your production. Next, I want to mention Next, I want to mention reliability. So, reliability. So, reliability. So, speed alone is speed alone is speed alone is not enough. So not enough. So not enough. So the reliability of AI the reliability of AI applications in applications in applications in production is production is production is part of speed. part of speed. So if an API is fast So if an API is fast on average but on average but on average but frequently throws timeouts frequently throws timeouts , 5XX errors, , 5XX errors, , 5XX errors, 429 restrictions, or 429 restrictions, or 429 restrictions, or similar failures, the similar failures, the similar failures, the user user user experience is still experience is still experience is still ruined. This becomes even more ruined. This becomes even more ruined. This becomes even more important for important for important for agent agent agent workloads. As you workloads. As you workloads. As you know, an agent may know, an agent may know, an agent may need multiple need multiple need multiple model calls to model calls to model calls to complete a single complete a single complete a single task, and if even task, and if even task, and if even one call one call one call in the middle fails, the in the middle fails, the in the middle fails, the entire workflow entire workflow entire workflow may be disrupted. may be disrupted. may be disrupted. That's why we That's why we That's why we optimize more than just optimize more than just optimize more than just latency or latency or latency or bandwidth. bandwidth. bandwidth. We optimize the entire We optimize the entire We optimize the entire API experience, API experience, API experience, including including including error rate, error rate, error rate, uptime, uptime, uptime, and all of these aspects and all of these aspects and all of these aspects together. This OpenAI together. This OpenAI together. This OpenAI for Azure GLM 5.1 Pro snapshot shows for Azure GLM 5.1 Pro snapshot shows that Friendly AI is not only that Friendly AI is not only that Friendly AI is not only fast, but also fast, but also fast, but also extremely reliable extremely reliable , with an uptime of , with an uptime of over 99.9%. So, what I'm over 99.9%. So, what I'm saying is that a lot of saying is that a lot of inference providers here, of course , and at this event , and at this event , and at this event as well, will tell you that as well, will tell you that as well, will tell you that they're fast, and to some they're fast, and to some they're fast, and to some extent they are, but on extent they are, but on extent they are, but on certain tests under certain tests under certain tests under certain conditions. But for certain conditions. But for certain conditions. But for your application, the your application, the your application, the only test that only test that only test that really matters really matters really matters is your own is your own is your own workload. That is,

workload. That is, workload. That is, your prompts, your prompts, your prompts, token length, token length, token length, traffic structure, and your traffic structure, and your latency expectations. So, our latency expectations. So, our recommendation is simple. recommendation is simple. recommendation is simple. Please bring Please bring Please bring your load to your load to your load to us and test the us and test the us and test the full experience of working full experience of working full experience of working with us in terms of speed with us in terms of speed , reliability, , reliability, , reliability, cost and quality. And cost and quality. And cost and quality. And of course, we built of course, we built of course, we built our inference service our inference service our inference service specifically for these specifically for these specifically for these workloads workloads workloads and believe we can and believe we can and believe we can provide you with the best provide you with the best provide you with the best experience. We have a booth experience. We have a booth experience. We have a booth at this event, so at this event, so at this event, so come with your come with your come with your most difficult most difficult most difficult challenges and let's challenges and let's challenges and let's solve them together. solve them together. solve them together. Finally, I want to Finally, I want to Finally, I want to share a review share a review share a review from one of our from one of our from one of our clients—Kilocode. Kilocode clients—Kilocode. Kilocode is one of the is one of the is one of the largest platforms largest platforms largest platforms for for for programming agents, programming agents, programming agents, processing several processing several processing several trillion tokens trillion tokens trillion tokens every week. They every week. They every week. They have tested many have tested many have tested many different different different inference providers over inference providers over inference providers over the past few years, and the past few years, and the past few years, and during a split test for during a split test for during a split test for GLM models, they were GLM models, they were GLM models, they were particularly impressed with particularly impressed with particularly impressed with Friendly AI. Compared to Friendly AI. Compared to Friendly AI. Compared to many other many other many other providers, Friendly AI providers, Friendly AI providers, Friendly AI was over seven times was over seven times was over seven times faster and had a significantly faster and had a significantly faster and had a significantly lower lower lower error rate. So I believe error rate. So I believe error rate. So I believe you can you can you can have the same have the same have the same experience with Friendly AI. So, experience with Friendly AI. So, experience with Friendly AI. So, if you're building if you're building if you're building your own advanced your own advanced your own advanced open models, open models, open models, building on them building on them building on them , come , come , come try them out on Friendly try them out on Friendly try them out on Friendly AI, bring your AI, bring your AI, bring your real-world work real-world work real-world work tasks, and we'll tasks, and we'll tasks, and we'll show you what show you what show you what faster, faster, faster, easier inference can do. So easier inference can do. So , this is the end of my , this is the end of my

, this is the end of my talk, and if you talk, and if you talk, and if you have any questions, have any questions, have any questions, please ask them. please ask them. please ask them. Thank you.