Agents That Own Their Inference — Du'an Lightfoot & Khaja Omer, Akamai Technologies

I've already mentioned that my name is I've already mentioned that my name is De'Juan Lightfoot. I'm a De'Juan Lightfoot. I'm a De'Juan Lightfoot. I'm a developer advocate at Akamai, and this is my developer advocate at Akamai, and this is my developer advocate at Akamai, and this is my colleague, colleague, colleague, Khadija Omar. I use the name Omar. Khadija Omar. I use the name Omar. Khadija Omar. I use the name Omar. I am a senior I am a senior I am a senior software engineer at Akamai. We'll be software engineer at Akamai. We'll be software engineer at Akamai. We'll be talking about agents that have their talking about agents that have their talking about agents that have their own reasoning, building own reasoning, building own reasoning, building AI agents for production on dedicated AI agents for production on dedicated AI agents for production on dedicated GPUs. Why GPUs. Why GPUs. Why are we here? Things are changing a bit are we here? Things are changing a bit are we here? Things are changing a bit . I started building . I started building . I started building agents before invoking tools. agents before invoking tools. In January 2023, I built In January 2023, I built my first agent. I would take the my first agent. I would take the my first agent. I would take the prompt, the context was then prompt, the context was then prompt, the context was then about 8000 tokens, parse it, about 8000 tokens, parse it, about 8000 tokens, parse it, then call the then call the then call the Python tool and return the response to LLM Python tool and return the response to LLM . Correct? But while I was doing it, . Correct? But while I was doing it, . Correct? But while I was doing it, I discovered something interesting. This I discovered something interesting. This I discovered something interesting. This can be really expensive. can be really expensive. can be really expensive. Correct? And then I started using Correct? And then I started using Correct? And then I started using local models to figure out, local models to figure out, local models to figure out, okay, how do I do this? And then okay, how do I do this? And then okay, how do I do this? And then I learned something else. This I learned something else. This I learned something else. This brings another challenge brings another challenge brings another challenge because if I use a smaller because if I use a smaller because if I use a smaller model, it may not be as efficient.

model, it may not be as efficient. model, it may not be as efficient. And then I may be limited by And then I may be limited by And then I may be limited by computing power. The memory in this GPU may computing power. The memory in this GPU may computing power. The memory in this GPU may not support the broader context of not support the broader context of not support the broader context of input and input and input and output data. And then multiple output data. And then multiple output data. And then multiple calls to tools. How can you best calls to tools. How can you best calls to tools. How can you best leverage GPUs, costs, and latency leverage GPUs, costs, and latency leverage GPUs, costs, and latency for production or just your for production or just your for production or just your own applications? Correct? own applications? Correct? own applications? Correct? That's why we're here – That's why we're here – we'll talk about agents who we'll talk about agents who we'll talk about agents who often rent out their intelligence often rent out their intelligence , and then how you can , and then how you can , and then how you can take back control and take it back. In take back control and take it back. In take back control and take it back. In the case of an agent, it's just a loop. the case of an agent, it's just a loop. the case of an agent, it's just a loop. You are calling LLM. It has access to You are calling LLM. It has access to You are calling LLM. It has access to tools, it has access to the environment, it tools, it has access to the environment, it tools, it has access to the environment, it makes decisions, reasoning about what makes decisions, reasoning about what makes decisions, reasoning about what actions to take when it is called actions to take when it is called actions to take when it is called or when it receives a prompt. or when it receives a prompt. or when it receives a prompt. These actions are then performed by These actions are then performed by These actions are then performed by tools, Python code, other tools, Python code, other tools, Python code, other programming languages, and then programming languages, and then programming languages, and then passed back to the model. passed back to the model. The model evaluates actions, results, and The model evaluates actions, results, and then updates the plan, takes then updates the plan, takes then updates the plan, takes additional actions, or the goal is additional actions, or the goal is additional actions, or the goal is achieved. True? But all this achieved. True? But all this requires generating tokens, right?

requires generating tokens, right? And depending on the application, this can And depending on the application, this can And depending on the application, this can lead to lead to lead to latency issues. So we'll talk about latency issues. So we'll talk about latency issues. So we'll talk about how you can manage all of this how you can manage all of this how you can manage all of this on dedicated graphics processing units on dedicated graphics processing units on dedicated graphics processing units (GPUs). What does it mean to (GPUs). What does it mean to (GPUs). What does it mean to own an inference? own an inference? own an inference? First, when it comes to owning First, when it comes to owning First, when it comes to owning inference, you actually inference, you actually inference, you actually own or lease the own or lease the own or lease the graphics processing unit (GPU). This means graphics processing unit (GPU). This means you have access to the you have access to the you have access to the graphics processing unit (GPU), the hardware that graphics processing unit (GPU), the hardware that graphics processing unit (GPU), the hardware that runs the AI, runs the AI, runs the AI, that runs the models, which means that runs the models, which means that runs the models, which means you have control over where the you have control over where the you have control over where the data goes, so it never data goes, so it never data goes, so it never leaves the cluster, the data center, or leaves the cluster, the data center, or leaves the cluster, the data center, or your laptop. True? Now you your laptop. True? Now you your laptop. True? Now you own this intelligence and what own this intelligence and what own this intelligence and what happens to the model. And if happens to the model. And if happens to the model. And if you're running in the cloud, you could also you're running in the cloud, you could also you're running in the cloud, you could also do something like private do something like private do something like private VPC endpoints or have it in a VPC VPC endpoints or have it in a VPC so you never leave it so you never leave it , right? I know we all have , right? I know we all have , right? I know we all have different environments, different ways of different environments, different ways of different environments, different ways of configuring our infrastructure, we can configuring our infrastructure, we can configuring our infrastructure, we can talk about all of that. Now, talk about all of that. Now, talk about all of that. Now, what's the difference? When do you what's the difference? When do you what's the difference? When do you own and when do you rent? Well own and when do you rent? Well own and when do you rent? Well , first of all, if your , first of all, if your , first of all, if your consumption is high and stable, consumption is high and stable, consumption is high and stable, why? Yes. Yes, at the end of why? Yes. Yes, at the end of why? Yes. Yes, at the end of the presentation I will provide a link to the slide and the presentation I will provide a link to the slide and the presentation I will provide a link to the slide and everything we showed, the code, everything we showed, the code, everything we showed, the code, everything, you can download it and everything, you can download it and everything, you can download it and take it with you. Thank you. And now that take it with you. Thank you. And now that take it with you. Thank you. And now that you are the owner? If you are the owner? If you are the owner? If the consumption is high, right? If you are the consumption is high, right? If you are the consumption is high, right? If you are constantly under load. Because if

constantly under load. Because if constantly under load. Because if you pay for a GPU, you are renting a GPU, what does that you pay for a GPU, you are renting a GPU, what does that you pay for a GPU, you are renting a GPU, what does that mean? This GPU should always mean? This GPU should always mean? This GPU should always work. Otherwise, what are work. Otherwise, what are work. Otherwise, what are you paying for? So this is something you you paying for? So this is something you you paying for? So this is something you need to think about. Then, when it need to think about. Then, when it need to think about. Then, when it comes to data, if you need comes to data, if you need comes to data, if you need that data to remain private, if you that data to remain private, if you that data to remain private, if you need to own it, you care about need to own it, you care about need to own it, you care about your intellectual property, or your intellectual property, or your intellectual property, or you have some you have some you have some governance and compliance reasons why you ca governance and compliance reasons why you ca governance and compliance reasons why you ca n't use a n't use a n't use a public public public AI provider, that's another AI provider, that's another AI provider, that's another reason to reason to reason to own it. And then the equipment. This is something we own it. And then the equipment. This is something we own it. And then the equipment. This is something we need to think about. Your hardware, need to think about. Your hardware, need to think about. Your hardware, if you can afford the if you can afford the if you can afford the capacity and you have the capacity, capacity and you have the capacity, capacity and you have the capacity, why not run these models, why not run these models, why not run these models, right? For your use cases. right? For your use cases. right? For your use cases. Because then you have full control. Because then you have full control. Because then you have full control. Now that you're renting? If your Now that you're renting? If your Now that you're renting? If your traffic is low, bursty, and traffic is low, bursty, and traffic is low, bursty, and you only need intelligence you only need intelligence you only need intelligence occasionally, having some occasionally, having some occasionally, having some GPUs may not be GPUs may not be GPUs may not be effective for you. Then, well effective for you. Then, well , we start talking about intelligence, , we start talking about intelligence, , we start talking about intelligence, more efficient models will more efficient models will more efficient models will cost those GPUs. And cost those GPUs. And cost those GPUs. And oftentimes, these vendors will provide that oftentimes, these vendors will provide that oftentimes, these vendors will provide that intelligence on intelligence on intelligence on GPUs because they have the infrastructure to do so GPUs because they have the infrastructure to do so GPUs because they have the infrastructure to do so . So it might make . So it might make . So it might make more sense to use those more sense to use those more sense to use those providers to get that providers to get that providers to get that intelligence, to get those benefits. And intelligence, to get those benefits. And intelligence, to get those benefits. And then, another part that's often then, another part that's often then, another part that's often challenging is that challenging is that challenging is that your teams may not have the knowledge your teams may not have the knowledge your teams may not have the knowledge or experience to run or experience to run or experience to run dedicated or self-hosted dedicated or self-hosted dedicated or self-hosted AI. And now AI. And now AI. And now today's plan. We'll start by today's plan. We'll start by today's plan. We'll start by covering the basics

covering the basics covering the basics you need to understand, such as the you need to understand, such as the you need to understand, such as the infrastructure stack we'll be infrastructure stack we'll be infrastructure stack we'll be using, the units, and the memory budget. We'll using, the units, and the memory budget. We'll using, the units, and the memory budget. We'll take a look at prefill and take a look at prefill and take a look at prefill and KB cache decoding and what KB cache decoding and what KB cache decoding and what it looks like. Next, we'll look at the it looks like. Next, we'll look at the it looks like. Next, we'll look at the dense, dense models compared dense, dense models compared dense, dense models compared to the expert mix. So you understand what to the expert mix. So you understand what to the expert mix. So you understand what it actually looks like on the it actually looks like on the it actually looks like on the GPU and why it's worth choosing one or the GPU and why it's worth choosing one or the GPU and why it's worth choosing one or the other. Omar will then discuss how to other. Omar will then discuss how to other. Omar will then discuss how to optimize inference. optimize inference. optimize inference. Correct? Once you understand that, okay Correct? Once you understand that, okay , we want to self-host, how , we want to self-host, how , we want to self-host, how can we optimize this to can we optimize this to can we optimize this to best utilize our GPUs, the best utilize our GPUs, the inference engine we're running, and our entire workload . And then, finally, we'll take . And then, finally, we'll take . And then, finally, we'll take that agent and deploy it to our that agent and deploy it to our that agent and deploy it to our own dedicated cluster. With that own dedicated cluster. With that own dedicated cluster. With that in mind, I pass this on to you, in mind, I pass this on to you, in mind, I pass this on to you, Omar. Omar. Omar. All right. Thank you. We will then discuss All right. Thank you. We will then discuss the environment in which each of you the environment in which each of you will participate in this workshop. will participate in this workshop. We have a Kubernetes cluster with over 200 We have a Kubernetes cluster with over 200 GPU nodes. Each of you has one GPU nodes. Each of you has one GPU nodes. Each of you has one GPU. We've set it up so GPU. We've set it up so GPU. We've set it up so that each of you has your that each of you has your that each of you has your own namespace. This own namespace. This own namespace. This namespace will be where the namespace will be where the namespace will be where the Jupiter notebook will run, as well as the Jupiter notebook will run, as well as the Jupiter notebook will run, as well as the vLLM inference endpoint you vLLM inference endpoint you vLLM inference endpoint you will be experimenting with. Then will be experimenting with. Then will be experimenting with. Then we will deploy the agent there as well. The cool thing is we will deploy the agent there as well. The cool thing is we will deploy the agent there as well. The cool thing is that the vLLM endpoint that the vLLM endpoint that the vLLM endpoint that we're going to run in your

that we're going to run in your that we're going to run in your namespace is something that you're going to optimize namespace is something that you're going to optimize namespace is something that you're going to optimize within the first four modules. within the first four modules. within the first four modules. I think this will be quite I think this will be quite I think this will be quite interesting. We'll look at the different interesting. We'll look at the different interesting. We'll look at the different techniques we'll techniques we'll techniques we'll use there to do this, and we'll use there to do this, and we'll use there to do this, and we'll deploy this deploy this deploy this vLLM endpoint iteratively until we vLLM endpoint iteratively until we vLLM endpoint iteratively until we optimize it to increase optimize it to increase optimize it to increase throughput, reduce throughput, reduce throughput, reduce latency, and make sure latency, and make sure latency, and make sure we meet all our SLAs. Now we meet all our SLAs. Now we meet all our SLAs. Now I would like to discuss one more I would like to discuss one more I would like to discuss one more issue: graphics processors in LKE. At issue: graphics processors in LKE. At issue: graphics processors in LKE. At Akamai I work on Kubernetes. This is my Akamai I work on Kubernetes. This is my Akamai I work on Kubernetes. This is my specialty. If anyone has specialty. If anyone has specialty. If anyone has questions about Kubernetes, please questions about Kubernetes, please questions about Kubernetes, please ask anytime. However, I would like to ask anytime. However, I would like to ask anytime. However, I would like to talk about what talk about what talk about what Akamai offers. We will be using Akamai offers. We will be using Akamai offers. We will be using Ada RTX 4000 GPUs. We also have Ada RTX 4000 GPUs. We also have Ada RTX 4000 GPUs. We also have other GPUs available, but other GPUs available, but other GPUs available, but these are dedicated. Each GPU has 200uh these are dedicated. Each GPU has 200uh these are dedicated. Each GPU has 200uh Each GPU has 2GB uh VRAM with a Each GPU has 2GB uh VRAM with a Each GPU has 2GB uh VRAM with a memory bandwidth of 260GB per memory bandwidth of 260GB per memory bandwidth of 260GB per second, so decoding will be second, so decoding will be second, so decoding will be quite fast because of this. Uh we have quite fast because of this. Uh we have quite fast because of this. Uh we have high performance in TFLOPS, uh a large high performance in TFLOPS, uh a large high performance in TFLOPS, uh a large number of CUDA cores, so pre- number of CUDA cores, so pre- number of CUDA cores, so pre- filling will also be quite filling will also be quite filling will also be quite fast because of that. Uh, currently, in fast because of that. Uh, currently, in fast because of that. Uh, currently, in your workshop, we're going to be using a your workshop, we're going to be using a your workshop, we're going to be using a single GPU. But that doesn't mean single GPU. But that doesn't mean single GPU. But that doesn't mean that when you want to deploy it at the end that when you want to deploy it at the end that when you want to deploy it at the end yourself, you can have different yourself, you can have different yourself, you can have different plans that we also offer plans that we also offer plans that we also offer where you can have up to four GPUs where you can have up to four GPUs where you can have up to four GPUs in a node; so that's up to 80 GB of uh in a node; so that's up to 80 GB of uh in a node; so that's up to 80 GB of uh VRAM on a single node, so you can VRAM on a single node, so you can VRAM on a single node, so you can deploy uh decent sized uh models

deploy uh decent sized uh models deploy uh decent sized uh models there as well. Uh and uh there as well. Uh and uh there as well. Uh and uh everything is kind of everything is kind of everything is kind of coordinated by LKE, so coordinated by LKE, so coordinated by LKE, so configuring the node, uh implementing configuring the node, uh implementing configuring the node, uh implementing the drivers, uh configuring the drivers, uh configuring the drivers, uh configuring everything for you, that way everything for you, that way everything for you, that way you don't have to do anything manually. you don't have to do anything manually. you don't have to do anything manually. Everything works like one click, Everything works like one click, Everything works like one click, you click a button, everything is you click a button, everything is you click a button, everything is implemented and you can get started. Yes, for implemented and you can get started. Yes, for implemented and you can get started. Yes, for more information we also have a more information we also have a more information we also have a link here, and at the end you will definitely find link here, and at the end you will definitely find link here, and at the end you will definitely find all these slides, so you all these slides, so you all these slides, so you can go to the pages. And then, er can go to the pages. And then, er , I'll give the floor to Dwan. , I'll give the floor to Dwan. Okay, so for the Okay, so for the inference engine we're going to use VLLM. inference engine we're going to use VLLM. inference engine we're going to use VLLM. Some of you may have heard of Some of you may have heard of Some of you may have heard of Ollama, which is great for Ollama, which is great for Ollama, which is great for local use, but VL-LLM local use, but VL-LLM local use, but VL-LLM offers some benefits that are better offers some benefits that are better offers some benefits that are better suited for a suited for a suited for a production environment. One is production environment. One is production environment. One is continuous batch processing immediately continuous batch processing immediately continuous batch processing immediately after installation, and the other is after installation, and the other is after installation, and the other is attention paging. It also supports OpenAI-compatible APIs attention paging. It also supports OpenAI-compatible APIs attention paging. It also supports OpenAI-compatible APIs , so if you're using , so if you're using , so if you're using any SDK that supports any SDK that supports any SDK that supports OpenAI, you can use it with v l l l. This OpenAI, you can use it with v l l l. This also allows you to make some also allows you to make some really interesting tweaks really interesting tweaks really interesting tweaks when setting up inference for when setting up inference for when setting up inference for endpoints and applications.

endpoints and applications. I mentioned continuous I mentioned continuous batch processing. For those of you who don't batch processing. For those of you who don't batch processing. For those of you who don't know what this is or how it know what this is or how it know what this is or how it actually works, this happens during the actually works, this happens during the actually works, this happens during the pre-fill stage, right? pre-fill stage, right? pre-fill stage, right? The prompts come in and typically in the The prompts come in and typically in the The prompts come in and typically in the case of static case of static case of static batch processing, let's say you have a batch processing, let's say you have a batch processing, let's say you have a school bus full of kids. Well, to be able to school bus full of kids. Well, to be able to school bus full of kids. Well, to be able to drop off new children using drop off new children using drop off new children using static batch processing, static batch processing, static batch processing, you have to drop off each child one by you have to drop off each child one by you have to drop off each child one by one, and then when all the one, and then when all the one, and then when all the children are dropped off, you go back for the children are dropped off, you go back for the children are dropped off, you go back for the rest and start another batch rest and start another batch . However, with continuous . However, with continuous . However, with continuous batch processing, you can batch processing, you can batch processing, you can drop off and pick up your child at the drop off and pick up your child at the drop off and pick up your child at the same time. This is much more same time. This is much more same time. This is much more efficient in terms of KB caching efficient in terms of KB caching that occurs during the that occurs during the that occurs during the pre-fill stage. Now, or pre-fill stage. Now, or pre-fill stage. Now, or in terms of page attention, in terms of page attention, in terms of page attention, page attention means that when we think about the page attention means that when we think about the page attention means that when we think about the KB cache and what's KB cache and what's KB cache and what's actually happening, we're actually happening, we're actually happening, we're dealing with memory space. dealing with memory space. So a prompt appears, and the So a prompt appears, and the longest prompt takes up longest prompt takes up longest prompt takes up memory space. Without page attention, this memory space. Without page attention, this memory space. Without page attention, this memory space is allocated memory space is allocated memory space is allocated even for shorter prompts, right?

even for shorter prompts, right? even for shorter prompts, right? But with page serving, everything But with page serving, everything But with page serving, everything is now managed by K z v l is now managed by K z v l is now managed by K z v l l l, ensuring that requests are continuously processed, l l, ensuring that requests are continuously processed, l l, ensuring that requests are continuously processed, generated, and batched generated, and batched generated, and batched as they arrive. as they arrive. It has a scheduler that sort of It has a scheduler that sort of manages manages manages batch processing, but that makes it batch processing, but that makes it batch processing, but that makes it really efficient. There are other really efficient. There are other really efficient. There are other service engines that we will discuss service engines that we will discuss service engines that we will discuss in this workshop. With that in in this workshop. With that in in this workshop. With that in mind, some of the metrics you're mind, some of the metrics you're mind, some of the metrics you're going to be interested in are going to be interested in are going to be interested in are throughput, right? To understand throughput, right? To understand how your GPU works, it's time for the how your GPU works, it's time for the how your GPU works, it's time for the first token. Understand, okay, first token. Understand, okay, first token. Understand, okay, you send a prompt, how long does it take to you send a prompt, how long does it take to you send a prompt, how long does it take to recover the first token? recover the first token? recover the first token? Next up, this KV cache utilization Next up, this KV cache utilization Next up, this KV cache utilization , we'll look at those , we'll look at those , we'll look at those metrics and show you how to get metrics and show you how to get metrics and show you how to get all of that from vLLM and then from all of that from vLLM and then from all of that from vLLM and then from Q. These are some of the metrics Q. These are some of the metrics Q. These are some of the metrics that we'll see in this that we'll see in this that we'll see in this workshop that you'll be able to workshop that you'll be able to workshop that you'll be able to understand better. And with this in understand better. And with this in understand better. And with this in mind, we will gain access to the environment mind, we will gain access to the environment . If you can scan this . If you can scan this . If you can scan this QR code, get out your laptops if QR code, get out your laptops if QR code, get out your laptops if you want to follow along, if not, you want to follow along, if not, you want to follow along, if not, you'll be following our workshops. In you'll be following our workshops. In you'll be following our workshops. In this Jupiter notebook, we have nine this Jupiter notebook, we have nine this Jupiter notebook, we have nine different labs that different labs that different labs that we will cover, and all the resources and we will cover, and all the resources and we will cover, and all the resources and dependencies are loaded into that dependencies are loaded into that dependencies are loaded into that environment. If you want to review the environment. If you want to review the environment. If you want to review the code and try it out yourself at code and try it out yourself at code and try it out yourself at home, I have a public repository on the home, I have a public repository on the home, I have a public repository on the Akamai Developers website. I'll have

Akamai Developers website. I'll have Akamai Developers website. I'll have that link, along with the presentation and that link, along with the presentation and that link, along with the presentation and everything we shared, everything we shared, everything we shared, available with one QR code that available with one QR code that available with one QR code that you can download. And if you can download. And if you can download. And if you need me to create a short you need me to create a short you need me to create a short link, I will. Okay, link, I will. Okay, link, I will. Okay, speaking of which, we'll start with the first speaking of which, we'll start with the first speaking of which, we'll start with the first workshop module if everyone workshop module if everyone workshop module if everyone is connected. Yeah, it is connected. Yeah, it is connected. Yeah, it was a bit slow for me too, so was a bit slow for me too, so was a bit slow for me too, so I used my own I used my own I used my own link shortener because I did link shortener because I did link shortener because I did it really fast in Rust. So yes. it really fast in Rust. So yes. I hope if you have a hotspot I would use it, but we'll try to use it, but we'll try to use it, but we'll try to make the most of the make the most of the make the most of the experience. Sounds good? Great experience. Sounds good? Great . Okay, so in the first module . Okay, so in the first module . Okay, so in the first module we'll cover the inference stack. We will we'll cover the inference stack. We will we'll cover the inference stack. We will connect to our vLLM endpoint, connect to our vLLM endpoint, connect to our vLLM endpoint, understand it, see how it generates understand it, see how it generates understand it, see how it generates tokens. We'll also take a look at tokens. We'll also take a look at tokens. We'll also take a look at the structure of our first agent, the structure of our first agent, the structure of our first agent, a few things we need to know, a few things we need to know, a few things we need to know, we'll look at the metrics, and then we'll look at the metrics, and then we'll look at the metrics, and then we'll understand the model we're we'll understand the model we're we'll understand the model we're connecting to. The model we'll use connecting to. The model we'll use connecting to. The model we'll use will be the three-four-B quantum model will be the three-four-B quantum model , but we have several PVC-based models , but we have several PVC-based models that can be loaded and unloaded that can be loaded and unloaded that can be loaded and unloaded for this endpoint. Speaking of for this endpoint. Speaking of for this endpoint. Speaking of v l l m, one of the things that comes with v l l m, one of the things that comes with v l l m, one of the things that comes with it is that when

it is that when it is that when you serve an endpoint, it you serve an endpoint, it you serve an endpoint, it only has one model, right? So if only has one model, right? So if only has one model, right? So if you want to support two models, you want to support two models, you want to support two models, you'll need another you'll need another you'll need another v l l m endpoint, right? This means v l l m endpoint, right? This means v l l m endpoint, right? This means you may need a routing you may need a routing you may need a routing model, a gateway in front of it to model, a gateway in front of it to model, a gateway in front of it to handle this. Supports handle this. Supports handle this. Supports authentication but not authentication but not authentication but not authorization. So you might need a authorization. So you might need a authorization. So you might need a gate, right? The agent gateway is gate, right? The agent gateway is gate, right? The agent gateway is often Kserve, this is another one. Hmm, often Kserve, this is another one. Hmm, often Kserve, this is another one. Hmm, any others? any others? any others? There are l l m d, there are many There are l l m d, there are many There are l l m d, there are many open source tools that make this possible. open source tools that make this possible. open source tools that make this possible. But the main one is But the main one is But the main one is Kserve, which we use a lot. And Kserve, which we use a lot. And Kserve, which we use a lot. And it's pretty good at scaling, it's pretty good at scaling, it's pretty good at scaling, routing, and authentication. routing, and authentication. routing, and authentication. Correct. Another thing it Correct. Another thing it Correct. Another thing it has is bandwidth throttling, has is bandwidth throttling, has is bandwidth throttling, right? If you are processing requests, right? If you are processing requests, right? If you are processing requests, you may want to limit your you may want to limit your you may want to limit your bandwidth. So, again, bandwidth. So, again, bandwidth. So, again, you want to find a gateway to you want to find a gateway to you want to find a gateway to put in front of it to manage put in front of it to manage put in front of it to manage our requests and also our requests and also our requests and also handle routing. I mentioned that handle routing. I mentioned that handle routing. I mentioned that in terms of execution time, in terms of execution time, in terms of execution time, there are other service endpoints there are other service endpoints there are other service endpoints that are service execution times that are service execution times that are service execution times that you can actually use for the that you can actually use for the that you can actually use for the server. S G lang is one, the RT tensor server. S G lang is one, the RT tensor server. S G lang is one, the RT tensor is another and llama C P which is another and llama C P which is another and llama C P which we'll talk about in a little more we'll talk about in a little more we'll talk about in a little more detail later so you can detail later so you can detail later so you can understand it. In this environment, you are understand it. In this environment, you are understand it. In this environment, you are connected to a Jupiter notebook connected to a Jupiter notebook connected to a Jupiter notebook that is located in the that is located in the . You also have a VLLM server endpoint . You also have a VLLM server endpoint . You also have a VLLM server endpoint , which you'll see in a moment.

, which you'll see in a moment. , which you'll see in a moment. All this is in a separate All this is in a separate All this is in a separate pod. Then you'll have pod. Then you'll have pod. Then you'll have another pod, well, another pod that another pod, well, another pod that another pod, well, another pod that you can run for the agent, and you can run for the agent, and you can run for the agent, and we'll see that too. So let's move on to we'll see that too. So let's move on to we'll see that too. So let's move on to installing our dependencies. installing our dependencies. We will install our dependencies. I've kind of already done this We will install our dependencies. I've kind of already done this , , , so some of it is already so some of it is already so some of it is already loaded in my runtime loaded in my runtime loaded in my runtime . I thought, we don't . I thought, we don't . I thought, we don't know what will happen with the internet, right know what will happen with the internet, right ? But the good thing is that once ? But the good thing is that once ? But the good thing is that once a request is sent, it happens within the a request is sent, it happens within the a request is sent, it happens within the cluster. So when we send our cluster. So when we send our cluster. So when we send our first request to understand first request to understand first request to understand why we're configured in why we're configured in why we're configured in VLLM, you can notice the model that VLLM, you can notice the model that VLLM, you can notice the model that we have, the name of your namespace in the we have, the name of your namespace in the we have, the name of your namespace in the cluster. The kube configuration is cluster. The kube configuration is cluster. The kube configuration is only for your only for your only for your namespace. This way, namespace. This way, namespace. This way, no one can access anyone else's no one can access anyone else's no one can access anyone else's namespace in this namespace in this namespace in this environment. And then we don't have a set of environment. And then we don't have a set of environment. And then we don't have a set of API keys, but it can be set API keys, but it can be set API keys, but it can be set using VLLM. We will now be using an using VLLM. We will now be using an using VLLM. We will now be using an Open AI API compliant endpoint Open AI API compliant endpoint Open AI API compliant endpoint to be able to connect to the to be able to connect to the to be able to connect to the VLLM Q Quinn 3 4B, 4 billion VLLM Q Quinn 3 4B, 4 billion VLLM Q Quinn 3 4B, 4 billion parameter model using the API. I think parameter model using the API. I think parameter model using the API. I think we use it in FP8. This means that it we use it in FP8. This means that it we use it in FP8. This means that it actually has 8 billion actually has 8 billion actually has 8 billion parameters. Is that 8 billion parameters. Is that 8 billion parameters. Is that 8 billion parameters?

parameters? parameters? No, 4 billion parameters. No, 4 billion parameters. No, 4 billion parameters. 4 billion parameters. 4 billion parameters. 4 billion parameters. BF16 BF16 BF16 Oh, BF That's the point. So, Oh, BF That's the point. So, Oh, BF That's the point. So, we use BF16 for precision, so we use BF16 for precision, so we use BF16 for precision, so we multiply bits, bytes, or parameters we multiply bits, bytes, or parameters we multiply bits, bytes, or parameters by two. That's 4 billion, which by two. That's 4 billion, which by two. That's 4 billion, which means 8 gigabytes per GPU. Now means 8 gigabytes per GPU. Now means 8 gigabytes per GPU. Now when we call, we have received the response when we call, we have received the response when we call, we have received the response for the model and we can see where for the model and we can see where for the model and we can see where it was actually handled. This was all it was actually handled. This was all it was actually handled. This was all in our in our in our cluster. That was pretty quick, cluster. That was pretty quick, cluster. That was pretty quick, right? It is located in our cluster right? It is located in our cluster . So we hope that thanks to . So we hope that thanks to . So we hope that thanks to this experience we will not end up with a this experience we will not end up with a this experience we will not end up with a public API. We are already public API. We are already public API. We are already connected to this HTTP protocol, connected to this HTTP protocol, connected to this HTTP protocol, everything will happen everything will happen everything will happen directly in the cluster. The response directly in the cluster. The response directly in the cluster. The response should be a bit faster. Now at should be a bit faster. Now at should be a bit faster. Now at the end we will trace the middle trace so the end we will trace the middle trace so the end we will trace the middle trace so that we can understand what it that we can understand what it that we can understand what it looks like during the pre- looks like during the pre- looks like during the pre- padding, KV buffering and padding, KV buffering and padding, KV buffering and decoding phase. So we have a stream decoding phase. So we have a stream decoding phase. So we have a stream that we'll configure. We'll look at that we'll configure. We'll look at that we'll configure. We'll look at the fragments and do some calculations.

the fragments and do some calculations. the fragments and do some calculations. We will calculate the time to first opening We will calculate the time to first opening We will calculate the time to first opening to understand it better. to understand it better. to understand it better. We'll see how all this is We'll see how all this is We'll see how all this is generated in a moment. If we run it generated in a moment. If we run it generated in a moment. If we run it again, I ran it again. We'll again, I ran it again. We'll again, I ran it again. We'll go over here and run it. This go over here and run it. This go over here and run it. This may take some time. However, we should may take some time. However, we should may take some time. However, we should see that the time to the see that the time to the see that the time to the first token was 37 first token was 37 first token was 37 milliseconds. Time to first token, milliseconds. Time to first token, milliseconds. Time to first token, decoding phase, what does each decoding phase, what does each decoding phase, what does each token after time to first token mean, token after time to first token mean, token after time to first token mean, right? That will be the internal right? That will be the internal right? That will be the internal delay, right? This is the internal delay, right? This is the internal delay, right? This is the internal delay that occurs when these delay that occurs when these delay that occurs when these tokens are processed. We calculate this and tokens are processed. We calculate this and tokens are processed. We calculate this and get about 40 tokens per get about 40 tokens per get about 40 tokens per second. This will probably be second. This will probably be second. This will probably be our limit. But when we start talking our limit. But when we start talking our limit. But when we start talking about bandwidth, it about bandwidth, it about bandwidth, it changes a little bit because it's just one changes a little bit because it's just one changes a little bit because it's just one request. Let's imagine we have request. Let's imagine we have request. Let's imagine we have hundreds of requests or 16 requests, this hundreds of requests or 16 requests, this hundreds of requests or 16 requests, this throughput will be much throughput will be much throughput will be much higher, but they will still be higher, but they will still be higher, but they will still be limited to these 40 tokens per limited to these 40 tokens per limited to these 40 tokens per second, right? Or it could go down, second, right? Or it could go down, second, right? Or it could go down, but we could still get more out of but we could still get more out of but we could still get more out of that bandwidth. And I think that bandwidth. And I think that bandwidth. And I think Omar will delve into this more.

Omar will delve into this more. So one of the things I want So one of the things I want to show is why not just to show is why not just to show is why not just run it on the CPU run it on the CPU ? Hmm, I didn't want the processor ? Hmm, I didn't want the processor ? Hmm, I didn't want the processor to be environmental because the one under to be environmental because the one under to be environmental because the one under um for this container for the um for this container for the um for this container for the Jupiter notebook is only 1 GB, right? So Jupiter notebook is only 1 GB, right? So Jupiter notebook is only 1 GB, right? So trying to run the local model on trying to run the local model on trying to run the local model on this will just hang. So, this will just hang. So, this will just hang. So, I kind of simulated what it would I kind of simulated what it would I kind of simulated what it would look like, but if look like, but if look like, but if you're trying to run a local model on the you're trying to run a local model on the you're trying to run a local model on the CPU, it's going to be much slower, CPU, it's going to be much slower, CPU, it's going to be much slower, right? It cannot perform complex right? It cannot perform complex right? It cannot perform complex multiplication on these matrices like the GPU can multiplication on these matrices like the GPU can . This is the purpose of the GPU. It performs all . This is the purpose of the GPU. It performs all . This is the purpose of the GPU. It performs all these calculations in parallel, which makes them these calculations in parallel, which makes them these calculations in parallel, which makes them much faster, right? So much faster, right? So much faster, right? So we simulate it on the CPU. And then from there we simulate it on the CPU. And then from there we simulate it on the CPU. And then from there we actually look at it on the GPU to we actually look at it on the GPU to we actually look at it on the GPU to see what it looks like. see what it looks like. We got 45 tokens per second. And We got 45 tokens per second. And one of the things about this, and what's one of the things about this, and what's one of the things about this, and what's interesting to me, is that I do interesting to me, is that I do interesting to me, is that I do a lot of, I would say, local models a lot of, I would say, local models a lot of, I would say, local models and run them, but 45 tokens or 39 and run them, but 45 tokens or 39 and run them, but 45 tokens or 39 tokens per second is not that tokens per second is not that tokens per second is not that bad for an application if you're running bad for an application if you're running bad for an application if you're running it for a specific use case, it for a specific use case, it for a specific use case, right? If you have a specific right? If you have a specific right? If you have a specific use case and you don't care about use case and you don't care about use case and you don't care about latency, you don't care. For latency, you don't care. For latency, you don't care. For example, if you have a chatbot, you want example, if you have a chatbot, you want this type of first token to this type of first token to this type of first token to appear quickly. But in a appear quickly. But in a appear quickly. But in a standard app where

standard app where standard app where you just run a task in the background, those 39 you just run a task in the background, those 39 you just run a task in the background, those 39 tokens per second might not be tokens per second might not be tokens per second might not be that bad, right? So sometimes a that bad, right? So sometimes a that bad, right? So sometimes a lower tokens per second or a lower tokens per second or a lower tokens per second or a shorter time to first token shorter time to first token shorter time to first token might not be as important, might not be as important, might not be as important, depending on the application and the depending on the application and the depending on the application and the latency you care about, latency you care about, latency you care about, right? You may be more concerned right? You may be more concerned right? You may be more concerned with the intelligence of the task being performed with the intelligence of the task being performed with the intelligence of the task being performed than how quickly it actually than how quickly it actually than how quickly it actually happens, especially, as I mentioned, happens, especially, as I mentioned, happens, especially, as I mentioned, if it's a background process. Now we will build if it's a background process. Now we will build if it's a background process. Now we will build our first agent. Now for our first agent. Now for our first agent. Now for those of you who know that when those of you who know that when those of you who know that when we talk about agents, this agent doesn't have we talk about agents, this agent doesn't have we talk about agents, this agent doesn't have any tools. It's actually any tools. It's actually any tools. It's actually just a request, but it's the foundation of the agent, we just a request, but it's the foundation of the agent, we just a request, but it's the foundation of the agent, we give them a system prompt so they know give them a system prompt so they know give them a system prompt so they know their role when we their role when we their role when we give them the tools later in the course. But now we give them the tools later in the course. But now we give them the tools later in the course. But now we only give it one prompt and give only give it one prompt and give only give it one prompt and give it a system role as a solution architect agent it a system role as a solution architect agent it a system role as a solution architect agent for Akamai. And you can see that for Akamai. And you can see that for Akamai. And you can see that he was able to generate a response.

he was able to generate a response. This answer, quite quick, works. But then the question becomes, do But then the question becomes, do we want to do this at scale, or do we want to do this at scale, or do we want to do this at scale, or do we want to build an agent that we want to build an agent that we want to build an agent that does more than just take a does more than just take a does more than just take a prompt and generate a response, or might prompt and generate a response, or might prompt and generate a response, or might require a lot of tool calls, require a lot of tool calls, require a lot of tool calls, right? This is where we start right? This is where we start right? This is where we start thinking about things like thinking about things like thinking about things like KB cache. We start thinking about how much KB cache. We start thinking about how much context, because when we start context, because when we start context, because when we start talking about this KB cache, talking about this KB cache, talking about this KB cache, that's really how much context you can that's really how much context you can that's really how much context you can fit on this GPU, right? In fit on this GPU, right? In fit on this GPU, right? In continuous continuous continuous batch processing, you're sending a prompt to a model batch processing, you're sending a prompt to a model batch processing, you're sending a prompt to a model where there might be 100 users where there might be 100 users where there might be 100 users trying to send prompts trying to send prompts trying to send prompts at once, right? Well, continuous at once, right? Well, continuous at once, right? Well, continuous batch processing will batch processing will batch processing will vary in size. A smaller GPU can vary in size. A smaller GPU can vary in size. A smaller GPU can handle multiple requests if handle multiple requests if handle multiple requests if you use something like vLLM. you use something like vLLM. you use something like vLLM. Whereas if you're using something Whereas if you're using something Whereas if you're using something like Olama, you'll see that like Olama, you'll see that like Olama, you'll see that you'll be throttled much more quickly you'll be throttled much more quickly because it doesn't process these because it doesn't process these because it doesn't process these requests in the same way. But vLLM requests in the same way. But vLLM requests in the same way. But vLLM handles this much better handles this much better handles this much better thanks to its schedule.

thanks to its schedule. So let's configure our agent, So let's configure our agent, call it up, and take a look at the steps. The call it up, and take a look at the steps. The code we use is in code we use is in Common and you can see all the Common and you can see all the Common and you can see all the imported code that is imported code that is imported code that is custom. So we have a custom. So we have a custom. So we have a foundation.py file that allows us to foundation.py file that allows us to foundation.py file that allows us to take measurements using take measurements using take measurements using Prometheus with VLL, right? We can Prometheus with VLL, right? We can Prometheus with VLL, right? We can view these metrics. Speaking of view these metrics. Speaking of view these metrics. Speaking of metrics, you can use Prometheus metrics, you can use Prometheus metrics, you can use Prometheus and Grafana, right? At home I always and Grafana, right? At home I always and Grafana, right? At home I always use Grafana. Also, when use Grafana. Also, when use Grafana. Also, when I create agents, besides just I create agents, besides just I create agents, besides just looking at metrics, one of the looking at metrics, one of the looking at metrics, one of the first things I always first things I always first things I always add to agents is add to agents is add to agents is observability. True? LangFuse is observability. True? LangFuse is observability. True? LangFuse is something I use often. How many of you have something I use often. How many of you have something I use often. How many of you have heard of LangFuse? There are heard of LangFuse? There are heard of LangFuse? There are other tools, but they are open other tools, but they are open other tools, but they are open source. They can also be run source. They can also be run source. They can also be run in a cluster, alongside agents. And then you have in a cluster, alongside agents. And then you have in a cluster, alongside agents. And then you have access to the fact that now your access to the fact that now your access to the fact that now your observation is also private, observation is also private, observation is also private, right? And you can see every step right? And you can see every step , every trace, every scope in your , every trace, every scope in your , every trace, every scope in your agent along with the number of tokens, what agent along with the number of tokens, what agent along with the number of tokens, what the latency was, not just for the the latency was, not just for the the latency was, not just for the first token time, but for every first token time, but for every first token time, but for every tool call, right? If tool call, right? If tool call, right? If you're trying to troubleshoot or you're trying to troubleshoot or you're trying to troubleshoot or you're trying to do some evaluation, now you're trying to do some evaluation, now you're trying to do some evaluation, now you can see all the you can see all the you can see all the delay steps and ranges, but then you delay steps and ranges, but then you delay steps and ranges, but then you can see, OK, I'm using a can see, OK, I'm using a can see, OK, I'm using a smaller model and it's making a lot smaller model and it's making a lot smaller model and it's making a lot more tool calls. This takes more tool calls. This takes more tool calls. This takes much longer than it should.

much longer than it should. much longer than it should. Why? Well, this is where Why? Well, this is where Why? Well, this is where ratings come in. Omar will talk ratings come in. Omar will talk ratings come in. Omar will talk more about this, but it's important to have more about this, but it's important to have more about this, but it's important to have visibility into your agents so you can visibility into your agents so you can visibility into your agents so you can understand how to optimize when understand how to optimize when understand how to optimize when you're using a smaller model. And that could you're using a smaller model. And that could you're using a smaller model. And that could be something as simple as a prompt, be something as simple as a prompt, be something as simple as a prompt, removing tools, or making the removing tools, or making the removing tools, or making the agent a little more agent a little more agent a little more specialized. And so we'll launch it. specialized. And so we'll launch it. And so, let's look at the steps. And so, let's look at the steps. We will look at the maximum number of We will look at the maximum number of We will look at the maximum number of tokens. We should see the time to the tokens. We should see the time to the tokens. We should see the time to the first token. So you can see that first token. So you can see that first token. So you can see that as the number of tokens increases, what as the number of tokens increases, what as the number of tokens increases, what happens to the first token time happens to the first token time happens to the first token time in millimilliseconds as we increase in millimilliseconds as we increase in millimilliseconds as we increase the number of prompts, right? As the number of prompts, right? As the number of prompts, right? As the number of prompts increases, the the number of prompts increases, the the number of prompts increases, the first token time increases. first token time increases. first token time increases. True? The question is: what True? The question is: what True? The question is: what do we optimize on? Do we optimize on the do we optimize on? Do we optimize on the do we optimize on? Do we optimize on the median or the largest? True? The one that median or the largest? True? The one that median or the largest? True? The one that took the longest took the longest took the longest during the delay. Well, during the delay. Well, during the delay. Well, we optimize on the one that takes we optimize on the one that takes we optimize on the one that takes the most time, not the median, the most time, not the median, the most time, not the median, right? This way we can right? This way we can right? This way we can increase the median. Now increase the median. Now increase the median. Now we'll talk about renting versus we'll talk about renting versus we'll talk about renting versus owning. Let's take a look at tokens.

owning. Let's take a look at tokens. owning. Let's take a look at tokens. How many are generated? I mentioned How many are generated? I mentioned that if you're using that if you're using that if you're using burst processing where you're not calling the burst processing where you're not calling the burst processing where you're not calling the model as often and you have a provider, model as often and you have a provider, model as often and you have a provider, versus an agent or versus an agent or versus an agent or agent apps that are always running, agent apps that are always running, agent apps that are always running, you can compare costs. you can compare costs. So we generated a response just So we generated a response just to get these tokens. Now let's take a to get these tokens. Now let's take a to get these tokens. Now let's take a look at the billing and look at the billing and look at the billing and compare which one will be compare which one will be compare which one will be more efficient. If you process, more efficient. If you process, more efficient. If you process, say, 200,000 tokens per day on a say, 200,000 tokens per day on a say, 200,000 tokens per day on a hosted and hosted and hosted and rented account, that would be around rented account, that would be around rented account, that would be around 1,500 per month. Correct? But 1,500 per month. Correct? But 1,500 per month. Correct? But if you rent GPU then it will be if you rent GPU then it will be if you rent GPU then it will be around 1000 per month. The size of the GPU, around 1000 per month. The size of the GPU, around 1000 per month. The size of the GPU, the size of the model, whether it actually performs the way the size of the model, whether it actually performs the way the size of the model, whether it actually performs the way you need it to does matter, you need it to does matter, you need it to does matter, but these are some of the things but these are some of the things but these are some of the things you think about when trying to you think about when trying to you think about when trying to compare whether to rent or buy. compare whether to rent or buy. compare whether to rent or buy. You can count it to You can count it to You can count it to break it down. We've already mentioned break it down. We've already mentioned break it down. We've already mentioned data resiliency, you know, compliance and data resiliency, you know, compliance and data resiliency, you know, compliance and security and control and all that security and control and all that security and control and all that good stuff when it comes to good stuff when it comes to good stuff when it comes to running on-premises. Before we get running on-premises. Before we get running on-premises. Before we get into the runtime, into the runtime, into the runtime, I mentioned that we use v l l, but I mentioned that we use v l l, but I mentioned that we use v l l, but you can also use SG lane, which you can also use SG lane, which you can also use SG lane, which is useful if you have requests is useful if you have requests is useful if you have requests that share long prefixes, that share long prefixes, that share long prefixes, multi-turn chats. Many organizations multi-turn chats. Many organizations multi-turn chats. Many organizations use SG lane for this purpose for agents.

use SG lane for this purpose for agents. use SG lane for this purpose for agents. As for the tensor, well, it's about the As for the tensor, well, it's about the As for the tensor, well, it's about the peak bandwidth of the peak bandwidth of the peak bandwidth of the video hardware. He truly specializes in video hardware. He truly specializes in video hardware. He truly specializes in providing this. And then providing this. And then providing this. And then llama.cpp for that edge. Such llama.cpp for that edge. Such llama.cpp for that edge. Such really fast processing. It can really fast processing. It can really fast processing. It can help with those highly help with those highly help with those highly quantized models directly on the quantized models directly on the quantized models directly on the ARM server. So we'll call it out ARM server. So we'll call it out ARM server. So we'll call it out again. And we look at our again. And we look at our again. And we look at our client just to make sure client just to make sure everything is still okay. everything is still okay. everything is still okay. We are checking if we can We are checking if we can We are checking if we can connect. We can see the connect. We can see the connect. We can see the supported model and we can see supported model and we can see supported model and we can see the ID. And I think I can the ID. And I think I can the ID. And I think I can print the whole thing. So we can print the whole thing. So we can print the whole thing. So we can see what this request actually see what this request actually see what this request actually looks like when it returns. So I'll print looks like when it returns. So I'll print looks like when it returns. So I'll print it out in full as well. So now it out in full as well. So now it out in full as well. So now we can see it and print it out we can see it and print it out we can see it and print it out so we can see it even so we can see it even so we can see it even better. And it's ready. When the request better. And it's ready. When the request better. And it's ready. When the request comes back, you will see the ID.

comes back, you will see the ID. comes back, you will see the ID. You can see the maximum You can see the maximum You can see the maximum length of the model, i.e. the length length of the model, i.e. the length length of the model, i.e. the length of the context, you can control it. of the context, you can control it. of the context, you can control it. True? You can of course adjust True? You can of course adjust True? You can of course adjust these parameters as you work with the model. these parameters as you work with the model. these parameters as you work with the model. But we can see everything about the But we can see everything about the But we can see everything about the model itself. Now we move on to the next model itself. Now we move on to the next model itself. Now we move on to the next session. This will require further session. This will require further session. This will require further development when you can development when you can development when you can explore the cluster, explore vLLM explore the cluster, explore vLLM . You can do it freely in this . You can do it freely in this notebook. So I'll close it, I'll abandon notebook. So I'll close it, I'll abandon it. And we'll move on to units and it. And we'll move on to units and it. And we'll move on to units and memory. Now, next session, and I memory. Now, next session, and I memory. Now, next session, and I 'm going to speed it up, so I'm working 'm going to speed it up, so I'm working 'm going to speed it up, so I'm working on it to give myself some time. Now on it to give myself some time. Now , when it comes to units and memory, , when it comes to units and memory, , when it comes to units and memory, we're going to talk about the ceiling in terms of we're going to talk about the ceiling in terms of we're going to talk about the ceiling in terms of tokens per second when tokens per second when tokens per second when using the model. We will also take a look at the using the model. We will also take a look at the using the model. We will also take a look at the actual actual actual model setup and understand what the model model setup and understand what the model model setup and understand what the model actually looks like and how it actually looks like and how it actually looks like and how it performs in the context of the GPU. As performs in the context of the GPU. As performs in the context of the GPU. As Omar mentioned, we use…and I'm going to Omar mentioned, we use…and I'm going to Omar mentioned, we use…and I'm going to scale that down a bit. Let's see if we can scale that down a bit. Let's see if we can scale that down a bit. Let's see if we can fit more in. Okay, fit more in. Okay, fit more in. Okay, so if we're talking about a model so if we're talking about a model so if we're talking about a model actually being placed on the GPU, we have actually being placed on the GPU, we have actually being placed on the GPU, we have model weights for a 4 billion model weights for a 4 billion model weights for a 4 billion parameter model, those model weights in BF16 parameter model, those model weights in BF16 parameter model, those model weights in BF16 will be 8 gigabytes. So we have 8 will be 8 gigabytes. So we have 8 will be 8 gigabytes. So we have 8 gigabytes and the rest of the space will be taken up by the gigabytes and the rest of the space will be taken up by the gigabytes and the rest of the space will be taken up by the KB cache. It's about KB cache. It's about KB cache. It's about how much context you can fit on that how much context you can fit on that how much context you can fit on that GPU because everything has to GPU because everything has to GPU because everything has to fit there. Now when a

fit there. Now when a fit there. Now when a prompt appears, when we talk about prefill, the model prompt appears, when we talk about prefill, the model prompt appears, when we talk about prefill, the model will read the entire prompt and store it in the will read the entire prompt and store it in the will read the entire prompt and store it in the KB cache. And then KB cache. And then KB cache. And then every time it generates a new every time it generates a new every time it generates a new token, it will update the prefill, so the token, it will update the prefill, so the token, it will update the prefill, so the prefill will grow. prefill will grow. prefill will grow. Correct? So, just Correct? So, just Correct? So, just because your prompt that is being sent because your prompt that is being sent because your prompt that is being sent is one size, as it is is one size, as it is is one size, as it is processed in the model, it will processed in the model, it will processed in the model, it will grow larger as it is generated. grow larger as it is generated. grow larger as it is generated. True? Now imagine the True? Now imagine the True? Now imagine the tool calls, the system prompt. They tool calls, the system prompt. They tool calls, the system prompt. They all go into the model and all go into the model and all go into the model and need to fit into the need to fit into the need to fit into the KB cache so that it can efficiently KB cache so that it can efficiently KB cache so that it can efficiently handle your request. handle your request. We also look at if We also look at if we're using FP8, which Omar will talk we're using FP8, which Omar will talk we're using FP8, which Omar will talk about quantization and what that actually looks like about quantization and what that actually looks like about quantization and what that actually looks like to shrink the model while still to shrink the model while still to shrink the model while still maintaining the expected maintaining the expected maintaining the expected performance. Now we will install our performance. Now we will install our performance. Now we will install our dependencies. And that's why we're going to connect here to dependencies. And that's why we're going to connect here to dependencies. And that's why we're going to connect here to verify what verify what verify what we've uploaded. So we have the we've uploaded. So we have the we've uploaded. So we have the Queen 3 model again with 4 billion Queen 3 model again with 4 billion Queen 3 model again with 4 billion parameters, our parameters, our parameters, our workshop namespace, cluster API key. Now, for workshop namespace, cluster API key. Now, for workshop namespace, cluster API key. Now, for models, we have the models, we have the models, we have the config.json file. What exactly is this config.json file. What exactly is this config.json file. What exactly is this file? After downloading the model, you receive a file? After downloading the model, you receive a file? After downloading the model, you receive a config.json file. Let's say config.json file. Let's say config.json file. Let's say you go to Hugging Face, download the you go to Hugging Face, download the you go to Hugging Face, download the model, you get a bunch of files.

model, you get a bunch of files. model, you get a bunch of files. All these files are needed, All these files are needed, All these files are needed, right? You will get a config.json file, right? You will get a config.json file, right? You will get a config.json file, you will get the saved tensor files. These you will get the saved tensor files. These you will get the saved tensor files. These tensor files are the model weights, right? tensor files are the model weights, right? tensor files are the model weights, right? But the configuration tells vLLM how But the configuration tells vLLM how But the configuration tells vLLM how to operate this model. True? So you can to operate this model. True? So you can to operate this model. True? So you can actually look at it and actually look at it and actually look at it and examine these files to understand what examine these files to understand what examine these files to understand what they actually look like. So here they actually look like. So here they actually look like. So here we show what this configuration file looks like we show what this configuration file looks like we show what this configuration file looks like . So we have 128 layers, . So we have 128 layers, . So we have 128 layers, size is 1024, tension head. size is 1024, tension head. size is 1024, tension head. There are 16 attention heads. The values ​​are There are 16 attention heads. The values ​​are There are 16 attention heads. The values ​​are eight, the dictionary size. True? How does eight, the dictionary size. True? How does eight, the dictionary size. True? How does the model train? What is the the model train? What is the the model train? What is the dictionary size of this model? True? You can dictionary size of this model? True? You can dictionary size of this model? True? You can actually look at these configurations actually look at these configurations . Can you modify No. True . Can you modify No. True ? These files cannot be modified, ? These files cannot be modified, ? These files cannot be modified, but they can be tuned. True? but they can be tuned. True? but they can be tuned. True? You can use lower. You can do You can use lower. You can do You can use lower. You can do all of these things to all of these things to all of these things to customize your model, but trying to customize your model, but trying to customize your model, but trying to modify these files will result in a modify these files will result in a modify these files will result in a crash. It won't load. Now crash. It won't load. Now crash. It won't load. Now we will look at the number of bytes per we will look at the number of bytes per we will look at the number of bytes per weight and its origin. We'll calculate.

weight and its origin. We'll calculate. weight and its origin. We'll calculate. Here we analyze a smaller model in BF16 Here we analyze a smaller model in BF16 Here we analyze a smaller model in BF16 for a model with 0.6 billion parameters. for a model with 0.6 billion parameters. for a model with 0.6 billion parameters. We multiply the parameters by two, i.e. 1.5 We multiply the parameters by two, i.e. 1.5 We multiply the parameters by two, i.e. 1.5 gigabytes. And for models with 8 billion gigabytes. And for models with 8 billion gigabytes. And for models with 8 billion parameters, what do these parameters, what do these parameters, what do these parameters look like, it will be 8 gigabytes. And parameters look like, it will be 8 gigabytes. And parameters look like, it will be 8 gigabytes. And if we compare BF16 with FP8. Now if we compare BF16 with FP8. Now if we compare BF16 with FP8. Now if we talk about precision, most if we talk about precision, most if we talk about precision, most models after training will models after training will models after training will sometimes have FP32 but we will often get it sometimes have FP32 but we will often get it sometimes have FP32 but we will often get it at FP16 for precision. This means that at FP16 for precision. This means that at FP16 for precision. This means that we multiply the model size by what? we multiply the model size by what? we multiply the model size by what? Four? For four, right? So this Four? For four, right? So this Four? For four, right? So this four billionth circuit would be four billionth circuit would be four billionth circuit would be 16 gigabytes at full precision, 16 gigabytes at full precision, 16 gigabytes at full precision, right? On your GPU. When right? On your GPU. When right? On your GPU. When you quantize it to, say, FP8, well you quantize it to, say, FP8, well , you lose some model capability, , you lose some model capability, , you lose some model capability, but not much, right? But then but not much, right? But then but not much, right? But then you start judging. True? If you have you start judging. True? If you have you start judging. True? If you have computational constraints, you have this computational constraints, you have this computational constraints, you have this larger model, maybe 70 billion larger model, maybe 70 billion larger model, maybe 70 billion parameters, that maybe you can parameters, that maybe you can parameters, that maybe you can do in FP8 or maybe with int4, right do in FP8 or maybe with int4, right ? Until the point where 70 ? Until the point where 70 ? Until the point where 70 billion model parameters are billion model parameters are billion model parameters are 16 gigabytes instead of 32.

16 gigabytes instead of 32. 16 gigabytes instead of 32. True? If possible, right? I don't True? If possible, right? I don't True? If possible, right? I don't know how much precision you'll lose, but know how much precision you'll lose, but know how much precision you'll lose, but you'll have to judge that. So you'll have to judge that. So , the difficulty of the assessment is paramount. , the difficulty of the assessment is paramount. So we look at the GPU where the So we look at the GPU where the weights are. So here you multiply BF16 weights are. So here you multiply BF16 weights are. So here you multiply BF16 by two, FP8 you multiply by one. by two, FP8 you multiply by one. by two, FP8 you multiply by one. We are looking at this for Quen. Then We are looking at this for Quen. Then We are looking at this for Quen. Then we have another model that we'll be we have another model that we'll be we have another model that we'll be using in this course where eight using in this course where eight using in this course where eight billion parameters is five billion parameters is five billion parameters is five gigabytes on the GPU, which allows you gigabytes on the GPU, which allows you gigabytes on the GPU, which allows you to see how much you can save to see how much you can save to see how much you can save with this intelligence with this intelligence with this intelligence working in the use cases we're discussing working in the use cases we're discussing working in the use cases we're discussing if you choose to if you choose to if you choose to down-quantize. down-quantize. down-quantize. So we'll take a look at this card, gigabytes per So we'll take a look at this card, gigabytes per So we'll take a look at this card, gigabytes per second. We'll take this where second. We'll take this where second. We'll take this where we're actually calculating, okay, we know the we're actually calculating, okay, we know the we're actually calculating, okay, we know the GPU speed is 360, and GPU speed is 360, and GPU speed is 360, and we show how to calculate to get we show how to calculate to get we show how to calculate to get tokens per second and where we're tokens per second and where we're tokens per second and where we're actually limited. So we will be actually limited. So we will be actually limited. So we will be limited by the actual limited by the actual limited by the actual speed of the GPU as to how fast it can speed of the GPU as to how fast it can speed of the GPU as to how fast it can run. And we show here run. And we show here run. And we show here the calculations for the KV cache. I'm trying to the calculations for the KV cache. I'm trying to the calculations for the KV cache. I'm trying to go a little faster go a little faster go a little faster so we can move on to quantization so we can move on to quantization so we can move on to quantization and optimization. But if and optimization. But if and optimization. But if you analyze it, you can read it you analyze it, you can read it you analyze it, you can read it and understand how the and understand how the and understand how the KB cache pattern works. Often we don't KB cache pattern works. Often we don't KB cache pattern works. Often we don't need to analyze and figure this out, need to analyze and figure this out, need to analyze and figure this out, but it will be necessary when but it will be necessary when but it will be necessary when you're trying to calculate, for example, you're trying to calculate, for example, you're trying to calculate, for example, you have 100 users you want you have 100 users you want you have 100 users you want to serve and you have a smaller

to serve and you have a smaller to serve and you have a smaller graphics processing unit (GPU). The question is: I have a graphics processing unit (GPU). The question is: I have a graphics processing unit (GPU). The question is: I have a smaller GPU. Let's say I smaller GPU. Let's say I give you a perfect example. give you a perfect example. give you a perfect example. I had to run a workshop and I had to run a workshop and I had to run a workshop and figure out how to handle 80 figure out how to handle 80 figure out how to handle 80 students and run students and run students and run inference for all the students inference for all the students inference for all the students with about three or with about three or with about three or four graphics processing units (GPUs) four graphics processing units (GPUs) four graphics processing units (GPUs) that had about 20 gigabytes, that had about 20 gigabytes, that had about 20 gigabytes, right? Do I need so many of them? right? Do I need so many of them? right? Do I need so many of them? Will this fit? Well, how Will this fit? Well, how Will this fit? Well, how big is my context for big is my context for big is my context for the load? What if everyone the load? What if everyone the load? What if everyone calls at the same time? Will calls at the same time? Will calls at the same time? Will GPUs or a cluster GPUs or a cluster GPUs or a cluster handle this? handle this? handle this? Will continuous batch processing work? So Will continuous batch processing work? So Will continuous batch processing work? So you configure the evaluation to evaluate you configure the evaluation to evaluate you configure the evaluation to evaluate the entire load and determine if it the entire load and determine if it the entire load and determine if it fits, if we hit that fits, if we hit that fits, if we hit that limit, right? Do we need to start limit, right? Do we need to start limit, right? Do we need to start shifting the load to the processor? shifting the load to the processor? shifting the load to the processor? Is your app slowing down? Okay, Is your app slowing down? Okay, Is your app slowing down? Okay, memory weight, memory budget memory weight, memory budget memory weight, memory budget versus KB pool. We'll take a look at versus KB pool. We'll take a look at versus KB pool. We'll take a look at this. Here's how you can actually this. Here's how you can actually this. Here's how you can actually calculate it. This is how we can get calculate it. This is how we can get calculate it. This is how we can get 37,000 tokens on this GPU, right? On a 37,000 tokens on this GPU, right? On a 37,000 tokens on this GPU, right? On a single GPU, we have 8 gigabytes for single GPU, we have 8 gigabytes for single GPU, we have 8 gigabytes for model weights, and then the space we model weights, and then the space we model weights, and then the space we have will be 37,000 have will be 37,000 have will be 37,000 tokens, right? So, if we have tokens, right? So, if we have tokens, right? So, if we have multiple users sending multiple users sending multiple users sending requests, right? Well, this will show you requests, right? Well, this will show you requests, right? Well, this will show you how many users and their how many users and their how many users and their context list. So we have four context list. So we have four context list. So we have four users who can send users who can send users who can send 8,000 tokens, right? For input, 8,000 tokens, right? For input, 8,000 tokens, right? For input, output, and tool calls, this will output, and tool calls, this will output, and tool calls, this will total 8000 tokens on this GPU. And total 8000 tokens on this GPU. And total 8000 tokens on this GPU. And now we show the tax. As

now we show the tax. As now we show the tax. As this increases, different requests and this increases, different requests and this increases, different requests and the generation of these requests. And it the generation of these requests. And it the generation of these requests. And it shows you how to do these calculations shows you how to do these calculations shows you how to do these calculations when you're trying to understand your when you're trying to understand your when you're trying to understand your compute limitations, your GPU, and the compute limitations, your GPU, and the compute limitations, your GPU, and the workloads you're running. So you can workloads you're running. So you can workloads you're running. So you can take all of that code, take all of that code, take all of that code, apply it, and analyze the apply it, and analyze the apply it, and analyze the use cases to see if it's a use cases to see if it's a use cases to see if it's a fit for GPUs. fit for GPUs. So I'll move on to some pre- So I'll move on to some pre- filling and decoding. When filling and decoding. When filling and decoding. When delving into pre- delving into pre- delving into pre- filling, decoding, and the filling, decoding, and the filling, decoding, and the KB cache, we're using the KB cache, we're using the KB cache, we're using the Qwen 3 model again, but we'll actually Qwen 3 model again, but we'll actually Qwen 3 model again, but we'll actually dig into the details to dig into the details to dig into the details to see what it looks like. So we have the see what it looks like. So we have the see what it looks like. So we have the initial seeding, which, as initial seeding, which, as initial seeding, which, as I mentioned, will I mentioned, will I mentioned, will focus on the time to first token, focus on the time to first token, focus on the time to first token, right? From the moment you hit the right? From the moment you hit the right? From the moment you hit the "Send" button, how long does it "Send" button, how long does it "Send" button, how long does it take to get the first token if take to get the first token if take to get the first token if you're streaming, right? Because you're streaming, right? Because you're streaming, right? Because if you're not streaming, if you're not streaming, if you're not streaming, you'll just get the whole response, you'll just get the whole response, you'll just get the whole response, but if you are streaming, but if you are streaming, but if you are streaming, how long does it take to get the first how long does it take to get the first how long does it take to get the first token back? True? And this will be the token back? True? And this will be the token back? True? And this will be the initial filling stage. Then initial filling stage. Then initial filling stage. Then the decoding will apply to this the decoding will apply to this the decoding will apply to this token, to each subsequent token.

token, to each subsequent token. token, to each subsequent token. Everything is measured. And if Everything is measured. And if Everything is measured. And if you look at the metrics in vLLM, send you look at the metrics in vLLM, send you look at the metrics in vLLM, send them to Grafana, you can display them them to Grafana, you can display them them to Grafana, you can display them on the dashboard and see how on the dashboard and see how on the dashboard and see how my prompts are increasing, you can my prompts are increasing, you can my prompts are increasing, you can see how they're increasing, how they're decreasing, see how they're increasing, how they're decreasing, see how they're increasing, how they're decreasing, right? True? And really understand right? True? And really understand right? True? And really understand what's actually happening on that GPU what's actually happening on that GPU when you're sending requests. We'll go through this when you're sending requests. We'll go through this when you're sending requests. We'll go through this , install our , install our , install our dependencies. Then we'll look at dependencies. Then we'll look at dependencies. Then we'll look at our environment, make sure our environment, make sure we're connected. Yes, and we're connected. Yes, and we're connected. Yes, and we still use Qwen 3 4B. We will look at we still use Qwen 3 4B. We will look at we still use Qwen 3 4B. We will look at self-attention, i.e. how self-attention, i.e. how self-attention, i.e. how is the KB cache processed is the KB cache processed is the KB cache processed ? This is the correct formula, so ? This is the correct formula, so ? This is the correct formula, so you can look at this softmax to you can look at this softmax to you can look at this softmax to really understand how the really understand how the really understand how the KB cache is handled in the KB cache is handled in the KB cache is handled in the model. We do it by hand. Just so you model. We do it by hand. Just so you can see the calculations on it can see the calculations on it can see the calculations on it . So, when it comes to . So, when it comes to . So, when it comes to KB cache, there is the KB cache, there is the KB cache, there is the naive approach and there is the naive approach and there is the naive approach and there is the KB cache approach that KB cache approach that KB cache approach that I would say is supported.

I would say is supported. I would say is supported. So the naive approach is that So the naive approach is that So the naive approach is that you take a prompt, the model parses the you take a prompt, the model parses the you take a prompt, the model parses the entire prompt, and then one by entire prompt, and then one by entire prompt, and then one by one it writes it to the one it writes it to the one it writes it to the KB cache and generates a response. The KB cache and generates a response. The KB cache says, "OK, KB cache says, "OK, I'll take the prompt, learn it, and I'll take the prompt, learn it, and I'll take the prompt, learn it, and save it, right? In parallel." save it, right? In parallel." So the KB cache allows you So the KB cache allows you to learn from the prompt to be able to to learn from the prompt to be able to to learn from the prompt to be able to reference the information, rather than just reference the information, rather than just reference the information, rather than just following a fixed following a fixed following a fixed sequence of naive sequence of naive sequence of naive caching approaches when it comes to handling the caching approaches when it comes to handling the caching approaches when it comes to handling the KB cache. If KB cache. If KB cache. If we generate this, we can see how it we generate this, we can see how it we generate this, we can see how it calculates the reduction by how much the calculates the reduction by how much the KB cache approach is used instead of the KB cache approach is used instead of the naive approach. We show it all naive approach. We show it all naive approach. We show it all manually and through code so you can manually and through code so you can manually and through code so you can really break it down really break it down really break it down and understand it. Now we're and understand it. Now we're and understand it. Now we're going to take a good look at going to take a good look at going to take a good look at KB caching usage and measure the time to KB caching usage and measure the time to KB caching usage and measure the time to first token. So we have 57 first token. So we have 57 first token. So we have 57 milliseconds, and the decoding took 25 milliseconds, and the decoding took 25 milliseconds, and the decoding took 25 milliseconds. We continue. We see what milliseconds. We continue. We see what milliseconds. We continue. We see what vLLM actually reveals, so we can vLLM actually reveals, so we can vLLM actually reveals, so we can take a closer look at these metrics and take a closer look at these metrics and take a closer look at these metrics and show some of them and show some of them and show some of them and how they are displayed in vLLM. Now we how they are displayed in vLLM. Now we how they are displayed in vLLM. Now we will call the prompt and increment its will call the prompt and increment its will call the prompt and increment its value, and you will be able to

value, and you will be able to value, and you will be able to watch the watch the watch the KB cache fill up. As you can see, this will take a KB cache fill up. As you can see, this will take a KB cache fill up. As you can see, this will take a while. So let's give him a few minutes for while. So let's give him a few minutes for while. So let's give him a few minutes for this process. We increase it to this process. We increase it to this process. We increase it to see the KB cache utilization see the KB cache utilization see the KB cache utilization and analyze and analyze the metrics. So we have the metrics. So we have peak usage running and then peak usage running and then peak usage running and then KB usage after it ends. KB usage after it ends. KB usage after it ends. This will be a mathematical calculation This will be a mathematical calculation This will be a mathematical calculation showing how you will process it. showing how you will process it. showing how you will process it. Again, the worker process Again, the worker process Again, the worker process we use in this code is located in the we use in this code is located in the we use in this code is located in the foundation.py file. So this will be a foundation.py file. So this will be a foundation.py file. So this will be a help for this code. Now we're going to look at help for this code. Now we're going to look at help for this code. Now we're going to look at warm, cold, um warm, cold, um warm, cold, um versus warm versus warm versus warm prefix caching. Now, prefix caching. Now, prefix caching. Now, prefix caching is an interesting thing. um don't confuse prefix caching is an interesting thing. um don't confuse prefix caching is an interesting thing. um don't confuse it with instant caching. And this it with instant caching. And this it with instant caching. And this is something I should have is something I should have is something I should have mentioned earlier. If mentioned earlier. If mentioned earlier. If you are using a provider and paying for a you are using a provider and paying for a you are using a provider and paying for a token, instant caching, token, instant caching, token, instant caching, especially if you have a continuous especially if you have a continuous especially if you have a continuous application running every 5 minutes, application running every 5 minutes, application running every 5 minutes, instant caching is instant caching is instant caching is a must. True?

a must. True? a must. True? You can reduce the cost by up to 75% You can reduce the cost by up to 75% You can reduce the cost by up to 75% depending on how you use it. depending on how you use it. depending on how you use it. So, there are several ways So, there are several ways So, there are several ways you can use a supplier and you can use a supplier and you can use a supplier and save on costs. But, when save on costs. But, when save on costs. But, when we talk about prefix caching, it we talk about prefix caching, it we talk about prefix caching, it is pre-caching that is pre-caching that is pre-caching that prefix for the KB buffer. And so, once we prefix for the KB buffer. And so, once we prefix for the KB buffer. And so, once we do that, we can see how it's do that, we can see how it's do that, we can see how it's actually handled here. actually handled here. actually handled here. So the code would be 220 milliseconds, So the code would be 220 milliseconds, So the code would be 220 milliseconds, and it dropped to 41 milliseconds. This all and it dropped to 41 milliseconds. This all and it dropped to 41 milliseconds. This all shows how to do it in code. shows how to do it in code. shows how to do it in code. So you can look it over. I'm trying to So you can look it over. I'm trying to So you can look it over. I'm trying to speed things up so Omar can speed things up so Omar can speed things up so Omar can get to his part and get to his part and get to his part and hopefully we can implement it. hopefully we can implement it. All right. So let's go back to the fourth All right. So let's go back to the fourth section where we will discuss the dense model section where we will discuss the dense model section where we will discuss the dense model versus the mixture of experts versus the mixture of experts versus the mixture of experts on GPU. Now, as for the on GPU. Now, as for the on GPU. Now, as for the dense model, what is it? Well, the dense model, what is it? Well, the dense model, what is it? Well, the dense model, when placed on the GPU, dense model, when placed on the GPU, is used to generate tokens, is used to generate tokens, right? So if it's 30 billion right? So if it's 30 billion right? So if it's 30 billion parameters, or let's use the model parameters, or let's use the model parameters, or let's use the model that we're using now. So Qwen 3 that we're using now. So Qwen 3 that we're using now. So Qwen 3 4B in BF16 8 8 GB on real GPU.

4B in BF16 8 8 GB on real GPU. Well, when it actually generates Well, when it actually generates tokens, when it actually works, tokens, when it actually works, tokens, when it actually works, this whole model works, right? So it this whole model works, right? So it this whole model works, right? So it will be a little slower than will be a little slower than will be a little slower than if you used the expert mix. if you used the expert mix. A mix of experts might be a A mix of experts might be a larger model, maybe 30 GB, but larger model, maybe 30 GB, but larger model, maybe 30 GB, but when it actually generates data within the when it actually generates data within the when it actually generates data within the model, it requires a group of experts, and only model, it requires a group of experts, and only model, it requires a group of experts, and only that group of experts does the that group of experts does the that group of experts does the generation. So even if it's generation. So even if it's generation. So even if it's 30GB, the generation itself might take 30GB, the generation itself might take 30GB, the generation itself might take 1.5GB. So it will be much faster 1.5GB. So it will be much faster . That's one of the advantages, even if . That's one of the advantages, even if . That's one of the advantages, even if it's a larger model, when it's a it's a larger model, when it's a it's a larger model, when it's a mix of experts, mix of experts, mix of experts, you can actually do it much better or you can actually do it much better or you can actually do it much better or much faster in the app. We're talking much faster in the app. We're talking much faster in the app. We're talking about memory here and why about memory here and why about memory here and why decoding is memory limited decoding is memory limited , meaning that when we're talking about the , meaning that when we're talking about the , meaning that when we're talking about the generation phase, after that time until the generation phase, after that time until the generation phase, after that time until the first token, when that generation first token, when that generation first token, when that generation actually happens, it's actually happens, it's actually happens, it's actually going to be memory limited. actually going to be memory limited. actually going to be memory limited. So, when we looked at the So, when we looked at the So, when we looked at the speed of this GPU, it is speed of this GPU, it is speed of this GPU, it is limited. There's no way to speed it up, limited. There's no way to speed it up, limited. There's no way to speed it up, right? There is no right? There is no right? There is no way to change this to make it work way to change this to make it work way to change this to make it work faster. But there are some things we faster. But there are some things we faster. But there are some things we can do to help this happen on a can do to help this happen on a can do to help this happen on a large scale. So we're going to take a look at the large scale. So we're going to take a look at the large scale. So we're going to take a look at the roof line and plot it on a graph so we roof line and plot it on a graph so we roof line and plot it on a graph so we can see it.

can see it. can see it. So let's make sure our model is So let's make sure our model is So let's make sure our model is loaded. So we had Qwen 3.4 loaded. So we had Qwen 3.4 loaded. So we had Qwen 3.4 billion parameters, right? And it will be billion parameters, right? And it will be billion parameters, right? And it will be computationally limited, so it will computationally limited, so it will computationally limited, so it will hit the roof line. So let's look at the hit the roof line. So let's look at the hit the roof line. So let's look at the flops. We'll look at this, flops. We'll look at this, flops. We'll look at this, so the top line is 297 flops per so the top line is 297 flops per so the top line is 297 flops per byte, right? And where it is byte, right? And where it is byte, right? And where it is limited by memory. Now, limited by memory. Now, limited by memory. Now, if we look at the mix of if we look at the mix of if we look at the mix of experts, now we're just going to look at it experts, now we're just going to look at it experts, now we're just going to look at it as code so we can as code so we can as code so we can actually understand it. So, with those actually understand it. So, with those actually understand it. So, with those 30 billion model parameters, 30 billion model parameters, 30 billion model parameters, right? 128 experts and only eight right? 128 experts and only eight right? 128 experts and only eight active per token. That's 48 layers, 2048 active per token. That's 48 layers, 2048 active per token. That's 48 layers, 2048 hidden. So now the mix of hidden. So now the mix of hidden. So now the mix of experts, these are the scales. This is the importance of experts, these are the scales. This is the importance of experts, these are the scales. This is the importance of participation. This is how much is used participation. This is how much is used participation. This is how much is used during generation. That's a lot, which during generation. That's a lot, which during generation. That's a lot, which would be much more efficient than would be much more efficient than would be much more efficient than using a smaller model. In using a smaller model. In using a smaller model. In fact, it may be faster when it fact, it may be faster when it fact, it may be faster when it actually generates. So here actually generates. So here actually generates. So here we're looking at a dense model, right? That's we're looking at a dense model, right? That's we're looking at a dense model, right? That's nine times fewer bytes than the nine times fewer bytes than the nine times fewer bytes than the dense model, right? So it decodes dense model, right? So it decodes dense model, right? So it decodes a little faster. It's kind of a little faster. It's kind of a little faster. It's kind of separating the two when separating the two when separating the two when you're trying to figure out if I want to use you're trying to figure out if I want to use you're trying to figure out if I want to use something like Mistral, right?

something like Mistral, right? something like Mistral, right? Or do I want to use a model like Or do I want to use a model like Or do I want to use a model like Qwen? We compare the sizes of the Qwen? We compare the sizes of the Qwen? We compare the sizes of the dense ones. So here we'll look at dense ones. So here we'll look at dense ones. So here we'll look at speed. Let's look at speed. Let's look at speed. Let's look at bandwidth, GPU server. bandwidth, GPU server. bandwidth, GPU server. Qwen 4B measured is about 40 tokens per Qwen 4B measured is about 40 tokens per Qwen 4B measured is about 40 tokens per second, right? The smaller Qwen model, second, right? The smaller Qwen model, second, right? The smaller Qwen model, 0.6, now it's not done on the 0.6, now it's not done on the 0.6, now it's not done on the CPU, but we're showing 300 tokens CPU, but we're showing 300 tokens CPU, but we're showing 300 tokens per second, right? But we're looking at a per second, right? But we're looking at a per second, right? But we're looking at a ceiling, we're actually looking at ceiling, we're actually looking at ceiling, we're actually looking at throughput and doing the math throughput and doing the math to get the maximum to get the maximum to get the maximum we can get, the ceiling we can get, the ceiling we can get, the ceiling we have on this GPU is 45 tokens per we have on this GPU is 45 tokens per we have on this GPU is 45 tokens per second. Now, if we enable second. Now, if we enable second. Now, if we enable inference in this model, we can inference in this model, we can inference in this model, we can analyze this. Now, when it comes to analyze this. Now, when it comes to analyze this. Now, when it comes to these Qwen models, if you want to use these Qwen models, if you want to use these Qwen models, if you want to use any type of agent and the model needs to any type of agent and the model needs to any type of agent and the model needs to infer, which means there might be a infer, which means there might be a infer, which means there might be a prompt, you have tools and it prompt, you have tools and it prompt, you have tools and it needs to make a lot of tool calls needs to make a lot of tool calls . Well, you'll have to engage in . Well, you'll have to engage in . Well, you'll have to engage in thinking. So now when it makes a thinking. So now when it makes a thinking. So now when it makes a tool call, if it gets the tool call, if it gets the tool call, if it gets the results and needs to make another results and needs to make another results and needs to make another tool call or verify the tool call or verify the tool call or verify the results to make them valid, you results to make them valid, you results to make them valid, you 're going to have to incorporate thinking 're going to have to incorporate thinking 're going to have to incorporate thinking into those models. And this is just a parameter into those models. And this is just a parameter into those models. And this is just a parameter that can be adjusted in the model, that can be adjusted in the model, that can be adjusted in the model, right? But with his thinking turned off, right? But with his thinking turned off, right? But with his thinking turned off, he generated a response that was he generated a response that was he generated a response that was 35 tokens. But with thinking 35 tokens. But with thinking , now you had to think about it and , now you had to think about it and , now you had to think about it and it cost about 600 tokens,

it cost about 600 tokens, it cost about 600 tokens, right? Now, with some right? Now, with some right? Now, with some models, like vendor models, models, like vendor models, models, like vendor models, you can actually set a you can actually set a you can actually set a token budget for that kind of thinking. So that's token budget for that kind of thinking. So that's token budget for that kind of thinking. So that's one of the things that you one of the things that you one of the things that you can save on costs can save on costs can save on costs with the vendor model. And perhaps with the vendor model. And perhaps with the vendor model. And perhaps some of the open models also some of the open models also some of the open models also allow this. I'm not sure. allow this. I'm not sure. allow this. I'm not sure. Yes, you can. Yes, you can. Yes, you can. Yes, you can also adjust this Yes, you can also adjust this Yes, you can also adjust this token budget to your thinking. So token budget to your thinking. So token budget to your thinking. So more tokens for thinking more tokens for thinking more tokens for thinking isn't always good, but sometimes it can isn't always good, but sometimes it can isn't always good, but sometimes it can help the model solve help the model solve help the model solve problems better and better achieve the problems better and better achieve the problems better and better achieve the expected end result. So expected end result. So expected end result. So when we talk about inference, there are when we talk about inference, there are when we talk about inference, there are a lot of things to consider, but one a lot of things to consider, but one a lot of things to consider, but one of the things we want to think about of the things we want to think about of the things we want to think about is the memory limit for the actual is the memory limit for the actual is the memory limit for the actual card and how to apply that to the card and how to apply that to the card and how to apply that to the agent. So we'll look at all agent. So we'll look at all agent. So we'll look at all the modules, we'll cover the modules, we'll cover the modules, we'll cover KB cache, prefix caching, KB cache, prefix caching, KB cache, prefix caching, attention heads, and also quantization, which attention heads, and also quantization, which attention heads, and also quantization, which Omar will cover in a moment. And then there's this Omar will cover in a moment. And then there's this Omar will cover in a moment. And then there's this continuous batch processing. continuous batch processing. continuous batch processing. We've discussed this before. If We've discussed this before. If We've discussed this before. If we separate prefill from decoding, we have a we separate prefill from decoding, we have a we separate prefill from decoding, we have a GPU prefill pool and decoding which GPU prefill pool and decoding which GPU prefill pool and decoding which is memory limited. So is memory limited. So is memory limited. So prefill is computationally limited, prefill is computationally limited, prefill is computationally limited, right? It has to go from memory to the right? It has to go from memory to the right? It has to go from memory to the GPU and then it has to be GPU and then it has to be GPU and then it has to be buffered, right? There is this phase, there are buffered, right? There is this phase, there are buffered, right? There is this phase, there are many optimizations that can be done many optimizations that can be done many optimizations that can be done , but when it comes to the , but when it comes to the , but when it comes to the decoding phase, there is nothing that decoding phase, there is nothing that decoding phase, there is nothing that can be done because we are can be done because we are can be done because we are limited by the GPU. That's why limited by the GPU. That's why limited by the GPU. That's why we're showing it in code instead of we're showing it in code instead of we're showing it in code instead of actually showing it because actually showing it because actually showing it because we're limited in what we can do in this we're limited in what we can do in this we're limited in what we can do in this workshop. With that in mind, I'll

workshop. With that in mind, I'll workshop. With that in mind, I'll make sure everything is in make sure everything is in make sure everything is in order. I don't think I need to order. I don't think I need to order. I don't think I need to change them. I think everything change them. I think everything change them. I think everything should be fine, Omar. should be fine, Omar. should be fine, Omar. I give you the floor. I give you the floor. I give you the floor. All right. All right. All right. And yes, we should start. You can And yes, we should start. You can And yes, we should start. You can talk about quantization. Thank you. talk about quantization. Thank you. talk about quantization. Thank you. Okay, thank you, DeJuan. Okay, thank you, DeJuan. Okay, thank you, DeJuan. Yes. Okay Yes. Okay , let me close this. , let me close this. Bright. Bright. Bright. It's actually yours. Here you go. It's actually yours. Here you go. It's actually yours. Here you go. Okay, great. So I'm going to talk about some Okay, great. So I'm going to talk about some Okay, great. So I'm going to talk about some completely different, um, completely different, um, completely different, um, optimization techniques that I think optimization techniques that I think optimization techniques that I think you can use to deploy these you can use to deploy these you can use to deploy these models on Kubernetes. As I said models on Kubernetes. As I said , my experience is with , my experience is with , my experience is with Kubernetes. I develop a lot of Kubernetes. I develop a lot of Kubernetes. I develop a lot of Kubernetes-related stuff at Akamai and Kubernetes-related stuff at Akamai and Kubernetes-related stuff at Akamai and work on their managed work on their managed work on their managed Kubernetes platform. So a lot of what we're Kubernetes platform. So a lot of what we're Kubernetes platform. So a lot of what we're doing is doing is doing is making sure that we're able to making sure that we're able to making sure that we're able to make good use of GPUs on Kubernetes, make good use of GPUs on Kubernetes, make good use of GPUs on Kubernetes, um, leverage all of their different um, leverage all of their different um, leverage all of their different features. Um, and uh, we implement a lot of features. Um, and uh, we implement a lot of features. Um, and uh, we implement a lot of these models using some these models using some these models using some of these techniques that I'll of these techniques that I'll of these techniques that I'll talk about. Well, there are different talk about. Well, there are different talk about. Well, there are different techniques. Let's talk about quantization.

techniques. Let's talk about quantization. techniques. Let's talk about quantization. I think this is the most important thing. I think this is the most important thing. I think this is the most important thing. It is super important. We'll talk about It is super important. We'll talk about It is super important. We'll talk about speculative decoding. This is a pretty speculative decoding. This is a pretty speculative decoding. This is a pretty cool technique. It can give you quite a cool technique. It can give you quite a cool technique. It can give you quite a big boost in performance and big boost in performance and big boost in performance and throughput. We'll talk about throughput. We'll talk about throughput. We'll talk about how to, you know, adjust your how to, you know, adjust your how to, you know, adjust your LLM and VLM configuration to make LLM and VLM configuration to make LLM and VLM configuration to make sure you're achieving maximum sure you're achieving maximum sure you're achieving maximum performance and meeting your SLAs, and performance and meeting your SLAs, and performance and meeting your SLAs, and then we'll do some tuning and then we'll do some tuning and then we'll do some tuning and evaluation. I'll leave you with some frameworks you evaluation. I'll leave you with some frameworks you evaluation. I'll leave you with some frameworks you can use to can use to can use to figure out what exact figure out what exact figure out what exact configuration you need for your configuration you need for your configuration you need for your specific workloads and specific specific workloads and specific specific workloads and specific use cases. So I'll focus on use cases. So I'll focus on use cases. So I'll focus on quantization. Let's see. And if we have quantization. Let's see. And if we have quantization. Let's see. And if we have time, we can certainly do a time, we can certainly do a time, we can certainly do a quick question and answer session, quick question and answer session, quick question and answer session, but I think we have 30 minutes. but I think we have 30 minutes. I'll discuss this at a good pace. I'll discuss this at a good pace. There are many different There are many different There are many different types of quantization in quantization. There is BF16, the model types of quantization in quantization. There is BF16, the model types of quantization in quantization. There is BF16, the model from which the model will start working. This is a from which the model will start working. This is a from which the model will start working. This is a full precision model full precision model full precision model provided by quantization. And there are provided by quantization. And there are provided by quantization. And there are many other quantizations besides. FP8 created many other quantizations besides. FP8 created . This is one of the models we will look at . This is one of the models we will look at . This is one of the models we will look at in this particular module. There is INT4 in this particular module. There is INT4 , AWQ, GPQ, GPTQ. These are different ways , AWQ, GPQ, GPTQ. These are different ways of quantizing the model. There's NVFP4, which is

Nvidia's proprietary quantization technique that they use. This is Nvidia's proprietary quantization technique that they use. This is hardware acceleration on hardware acceleration on hardware acceleration on Blackwell GPUs and beyond. This Blackwell GPUs and beyond. This Blackwell GPUs and beyond. This is pretty cool. But let's see. Let's is pretty cool. But let's see. Let's is pretty cool. But let's see. Let's dive into this. Let's do it. Let's dive into this. Let's do it. Let's hope we can hope we can fit it in there. All right. As you can see, fit it in there. All right. As you can see, fit it in there. All right. As you can see, you know, if you have BF16 loads, it you know, if you have BF16 loads, it you know, if you have BF16 loads, it will take up a lot of memory. In this will take up a lot of memory. In this will take up a lot of memory. In this particular use case we have 20 particular use case we have 20 particular use case we have 20 gigabytes of VRAM. And we want gigabytes of VRAM. And we want gigabytes of VRAM. And we want to maximize the number of requests to maximize the number of requests to maximize the number of requests we can handle with we can handle with we can handle with those 20 gigabytes. We want to those 20 gigabytes. We want to those 20 gigabytes. We want to make sure we have enough make sure we have enough make sure we have enough space for the KB cache. space for the KB cache. space for the KB cache. The reason is that the more The reason is that the more The reason is that the more requests you have, the more requests you have, the more requests you have, the more KB cache you can have in memory KB cache you can have in memory , the more requests you can , the more requests you can , the more requests you can process in parallel, process in parallel, process in parallel, thus increasing your throughput. So this is thus increasing your throughput. So this is thus increasing your throughput. So this is super important. That's why super important. That's why super important. That's why we want to perform quantization. We want we want to perform quantization. We want we want to perform quantization. We want to reduce the model size so that we to reduce the model size so that we to reduce the model size so that we can fit more can fit more can fit more KB cache and essentially KB cache and essentially KB cache and essentially increase throughput. Well, increase throughput. Well, increase throughput. Well, that's one of the reasons. The second important that's one of the reasons. The second important that's one of the reasons. The second important reason, I think, is also because reason, I think, is also because you're reducing the weights essentially by you're reducing the weights essentially by you're reducing the weights essentially by half, you don't have to do this half, you don't have to do this half, you don't have to do this memory transfer between different memory transfer between different memory transfer between different memory hierarchies, which results in memory hierarchies, which results in memory hierarchies, which results in essentially faster decoding.

essentially faster decoding. essentially faster decoding. So we'll see you go from, So we'll see you go from, So we'll see you go from, you know, 40 tokens per second, which is the you know, 40 tokens per second, which is the you know, 40 tokens per second, which is the baseline model, probably baseline model, probably baseline model, probably go up to about 60 tokens per go up to about 60 tokens per go up to about 60 tokens per second, if not more. Well, let's take a look at second, if not more. Well, let's take a look at second, if not more. Well, let's take a look at this. Let's install the this. Let's install the this. Let's install the dependencies. And then I'll make sure the dependencies. And then I'll make sure the dependencies. And then I'll make sure the model is still available and model is still available and model is still available and everything works. So yes, we have the everything works. So yes, we have the everything works. So yes, we have the model running. The same model with 4 model running. The same model with 4 model running. The same model with 4 billion parameters and runs in a billion parameters and runs in a billion parameters and runs in a namespace. Again, as namespace. Again, as namespace. Again, as I said, well, just a general I said, well, just a general I said, well, just a general overview, there's a whole other overview, there's a whole other overview, there's a whole other source of different techniques that you source of different techniques that you source of different techniques that you can use to do quantization. can use to do quantization. can use to do quantization. Um uh for example, like um uh Lama CBP Um uh for example, like um uh Lama CBP Um uh for example, like um uh Lama CBP is quite popular for uh is quite popular for uh is quite popular for uh running a lot of these models with running a lot of these models with running a lot of these models with lower precision. Uh uses the lower precision. Uh uses the lower precision. Uh uses the GGUF format. Uh this format is pretty cool GGUF format. Uh this format is pretty cool . You can have um uh you know, . You can have um uh you know, . You can have um uh you know, 5-bit, 3-bit, 2- 5-bit, 3-bit, 2- bit, 1-bit quantizations, which is pretty bit, 1-bit quantizations, which is pretty bit, 1-bit quantizations, which is pretty interesting. Um, and pretty cool. But interesting. Um, and pretty cool. But , the downside to that is you lose a lot of , the downside to that is you lose a lot of , the downside to that is you lose a lot of uh you lose quality trying to get a uh you lose quality trying to get a uh you lose quality trying to get a lot of performance out of it. So there is a lot of performance out of it. So there is a lot of performance out of it. So there is a compromise there. Um but, for the most part, compromise there. Um but, for the most part, compromise there. Um but, for the most part, going from BF16 to FP8 is a very going from BF16 to FP8 is a very going from BF16 to FP8 is a very slight drop in quality and slight drop in quality and slight drop in quality and performance. Um and this will result in performance. Um and this will result in performance. Um and this will result in greater efficiency while maintaining greater efficiency while maintaining greater efficiency while maintaining exactly the same quality. Uh so this is what exactly the same quality. Uh so this is what we want to show. This is we want to show. This is we want to show. This is one of the assumptions we make.

one of the assumptions we make. one of the assumptions we make. Obviously, when you do this for Obviously, when you do this for Obviously, when you do this for your specific workload, your specific workload, your specific workload, depending on what you're depending on what you're depending on what you're using it for, I highly encourage people to using it for, I highly encourage people to using it for, I highly encourage people to create their own evaluations. So create their own evaluations. So , that way, if you're coming from, , that way, if you're coming from, , that way, if you're coming from, you know, BF16, your evaluations are saying, you know, BF16, your evaluations are saying, you know, BF16, your evaluations are saying, "OK, everything's fine. "OK, everything's fine. "OK, everything's fine. Everything's working fine for our Everything's working fine for our Everything's working fine for our workload." When you switch to FP8, workload." When you switch to FP8, workload." When you switch to FP8, you might have issues where some you might have issues where some you might have issues where some things break, sometimes it's a things break, sometimes it's a things break, sometimes it's a hallucination, sometimes something loops hallucination, sometimes something loops . Many of these things can . Many of these things can . Many of these things can happen in very specific happen in very specific happen in very specific use cases, and in those use cases, and in those use cases, and in those scenarios, it's very important to scenarios, it's very important to scenarios, it's very important to have your own set of evaluations and have your own set of evaluations and have your own set of evaluations and run your own evaluations to run your own evaluations to run your own evaluations to make sure that any changes you make make sure that any changes you make make sure that any changes you make are allowed and don't cause are allowed and don't cause are allowed and don't cause performance degradation. Again, we'll performance degradation. Again, we'll performance degradation. Again, we'll run and make sure we have a run and make sure we have a run and make sure we have a BF16 database. And then we'll measure BF16 database. And then we'll measure BF16 database. And then we'll measure performance. So we have a small performance. So we have a small performance. So we have a small testing toolkit that we use testing toolkit that we use testing toolkit that we use for testing. Basically, if for testing. Basically, if for testing. Basically, if we look at the levels where they are like we look at the levels where they are like we look at the levels where they are like different concurrencies, we create this different concurrencies, we create this different concurrencies, we create this request. Each request will request. Each request will request. Each request will have approximately 256 tokens and the result have approximately 256 tokens and the result have approximately 256 tokens and the result will be approximately 128 tokens. So will be approximately 128 tokens. So , in a sense, we're trying to , in a sense, we're trying to , in a sense, we're trying to simulate how many requests can be simulate how many requests can be simulate how many requests can be processed simultaneously and processed simultaneously and processed simultaneously and we'll use that as a benchmark we'll use that as a benchmark we'll use that as a benchmark to see what the baseline performance is to see what the baseline performance is to see what the baseline performance is . So for 32, we get a . So for 32, we get a . So for 32, we get a total performance of 714 total performance of 714 total performance of 714 tokens per second, which is pretty tokens per second, which is pretty tokens per second, which is pretty good. With single good. With single good. With single concurrency we get 37.5

concurrency we get 37.5 concurrency we get 37.5 tokens per second. It's worth noting this tokens per second. It's worth noting this , and we also get 61 milliseconds for the , and we also get 61 milliseconds for the , and we also get 61 milliseconds for the time to the first token. And then time to the first token. And then time to the first token. And then you'll notice that the time to the first you'll notice that the time to the first you'll notice that the time to the first token increases, right? The jump between 32 token increases, right? The jump between 32 token increases, right? The jump between 32 and 64 is quite large. We go from and 64 is quite large. We go from and 64 is quite large. We go from 277 milliseconds to 1300 milliseconds. This is 277 milliseconds to 1300 milliseconds. This is 277 milliseconds to 1300 milliseconds. This is quite a big leap and it is not necessarily worth quite a big leap and it is not necessarily worth quite a big leap and it is not necessarily worth agreeing to such a compromise. This means it's agreeing to such a compromise. This means it's agreeing to such a compromise. This means it's potentially worth potentially worth potentially worth stopping at just 32 concurrent stopping at just 32 concurrent stopping at just 32 concurrent requests that you're processing and not requests that you're processing and not requests that you're processing and not going beyond that number. So it's worth going beyond that number. So it's worth going beyond that number. So it's worth noting. As I said, noting. As I said, noting. As I said, evaluations are extremely important. The reason is evaluations are extremely important. The reason is evaluations are extremely important. The reason is that during quantization there that during quantization there that during quantization there can be a drop in performance and a can be a drop in performance and a can be a drop in performance and a loss of intelligence. Reasoning may loss of intelligence. Reasoning may loss of intelligence. Reasoning may not work properly. There will be not work properly. There will be not work properly. There will be many problems. I'm basically simulating many problems. I'm basically simulating many problems. I'm basically simulating here at a very basic level here at a very basic level here at a very basic level what evaluations are worth considering.

what evaluations are worth considering. Consider whether the JSON is Consider whether the JSON is generated correctly or is not generated correctly or is not generated correctly or is not corrupted when corrupted when corrupted when dequantizing. In BF16 you may have good dequantizing. In BF16 you may have good dequantizing. In BF16 you may have good answers, but in FP8 answers, but in FP8 answers, but in FP8 function calls may fail. function calls may fail. Why? Probably because Why? Probably because the JSON is not generated correctly. the JSON is not generated correctly. the JSON is not generated correctly. There are many There are many There are many evaluation cases that already exist and evaluation cases that already exist and evaluation cases that already exist and can be used in can be used in can be used in open source software. Hugging Face has many open source software. Hugging Face has many open source software. Hugging Face has many such cases. You can find them such cases. You can find them such cases. You can find them on GitHub. But it's worth creating on GitHub. But it's worth creating on GitHub. But it's worth creating your own, appropriate for your specific your own, appropriate for your specific your own, appropriate for your specific use case. And then… So, the use case. And then… So, the use case. And then… So, the next thing we're going to do is next thing we're going to do is next thing we're going to do is transition from the BF16 model to the transition from the BF16 model to the transition from the BF16 model to the FP8 model. We're essentially going to reduce FP8 model. We're essentially going to reduce FP8 model. We're essentially going to reduce the size of the model by half, and when the size of the model by half, and when the size of the model by half, and when we do that, we'll see the we do that, we'll see the we do that, we'll see the performance differences and whether they're worth it, as well as performance differences and whether they're worth it, as well as performance differences and whether they're worth it, as well as whether we're losing quality. To whether we're losing quality. To whether we're losing quality. To do this, you need to go to the do this, you need to go to the do this, you need to go to the root directory, click "Manifest" and root directory, click "Manifest" and root directory, click "Manifest" and there you will see the vllm.yaml file. I will discuss there you will see the vllm.yaml file. I will discuss this. This is the typical manifest configuration file you this. This is the typical manifest configuration file you have for have for have for Kubernetes deployments. How many of you have used Kubernetes deployments. How many of you have used Kubernetes deployments. How many of you have used Kubernetes before and have knowledge of Kubernetes before and have knowledge of Kubernetes before and have knowledge of it? Okay, quite a lot.

it? Okay, quite a lot. it? Okay, quite a lot. Okay, great. This will be short. I Okay, great. This will be short. I Okay, great. This will be short. I don't need to go into all the don't need to go into all the don't need to go into all the details. But essentially we have a PVC details. But essentially we have a PVC details. But essentially we have a PVC that works, which is basically a that works, which is basically a that works, which is basically a cache where cache where cache where all our models live, all our models live, all our models live, and we have a simple implementation with vLLM with and we have a simple implementation with vLLM with and we have a simple implementation with vLLM with a lot of different arguments and flags a lot of different arguments and flags . So, you can see that we are using a . So, you can see that we are using a . So, you can see that we are using a Qwen model with 3-4 billion parameters. And Qwen model with 3-4 billion parameters. And Qwen model with 3-4 billion parameters. And then we'll switch it to the FP8 version then we'll switch it to the FP8 version then we'll switch it to the FP8 version from Red Hat. One more thing I from Red Hat. One more thing I from Red Hat. One more thing I want to quickly mention is the want to quickly mention is the want to quickly mention is the dynamics at the end. This is quite important. This dynamics at the end. This is quite important. This dynamics at the end. This is quite important. This means that not everything in this means that not everything in this means that not everything in this model is quantized to FP8. In model is quantized to FP8. In model is quantized to FP8. In many cases, with many different many cases, with many different many cases, with many different quantization techniques, you can quantization techniques, you can quantization techniques, you can quantize a model, just quantize a model, just quantize a model, just certain parts of it and not others, to certain parts of it and not others, to certain parts of it and not others, to make sure that the performance make sure that the performance make sure that the performance remains the same and there are not remains the same and there are not remains the same and there are not many problems there. That's one of the reasons why many problems there. That's one of the reasons why many problems there. That's one of the reasons why you're going to see dynamics, you're going to see dynamics, you're going to see dynamics, you're going to see a lot of different things.

you're going to see a lot of different things. you're going to see a lot of different things. You'll even see quantization where You'll even see quantization where You'll even see quantization where some model heads will be 4- some model heads will be 4- bit, others 8-bit, and still others bit, others 8-bit, and still others bit, others 8-bit, and still others full precision. And you know, it's full precision. And you know, it's full precision. And you know, it's quite interesting that people are quite interesting that people are quite interesting that people are doing this and seeing a big doing this and seeing a big doing this and seeing a big performance improvement there. They see an performance improvement there. They see an performance improvement there. They see an increase in efficiency increase in efficiency increase in efficiency while ensuring that quality while ensuring that quality while ensuring that quality remains the same. So, once this remains the same. So, once this remains the same. So, once this is changed, we'll save it and is changed, we'll save it and is changed, we'll save it and then quickly run it and then quickly run it and then quickly run it and apply it. The idea is to make sure apply it. The idea is to make sure apply it. The idea is to make sure the difference is correct. We're the difference is correct. We're the difference is correct. We're not causing any problems here. It not causing any problems here. It should be fine. And then should be fine. And then we just go ahead and where is the we just go ahead and where is the we just go ahead and where is the "Apply" button? Here. "Apply" button? Here. "Apply" button? Here. We just apply it and it will be implemented. This We just apply it and it will be implemented. This We just apply it and it will be implemented. This will take some time. Let me will take some time. Let me will take some time. Let me write this down. But another thing you write this down. But another thing you write this down. But another thing you can do is open a can do is open a can do is open a terminal tab. For example, terminal tab. For example, terminal tab. For example, if you click the plus sign, if you click the plus sign, if you click the plus sign, you'll see terminal options that you'll see terminal options that you'll see terminal options that let you see what's let you see what's let you see what's in your namespace and what you in your namespace and what you in your namespace and what you can do there. So if it can do there. So if it can do there. So if it shows up, you'll see that the shows up, you'll see that the shows up, you'll see that the VLLM endpoint keeps showing up. This pod is VLLM endpoint keeps showing up. This pod is VLLM endpoint keeps showing up. This pod is essentially your Jupiter notebook essentially your Jupiter notebook essentially your Jupiter notebook running in your namespace.

running in your namespace. Let's see. It will take some time for it to Let's see. It will take some time for it to appear. Yes. Since this is appear. Yes. Since this is appear. Yes. Since this is coming up, does anyone have any coming up, does anyone have any coming up, does anyone have any questions? I guess we can ask one questions? I guess we can ask one questions? I guess we can ask one question. question. It's pre-quantized, right? It's pre-quantized, right? What? What? What? The model we changed to was The model we changed to was The model we changed to was already pre-quantized. already pre-quantized. already pre-quantized. Yes, it was pre-quantized. Yes. Yes, it was pre-quantized. Yes. Yes, it was pre-quantized. Yes. You can quantize your own models. You can quantize your own models. You can quantize your own models. There is an LLM compressor. It's provided There is an LLM compressor. It's provided There is an LLM compressor. It's provided by VLLM if I remember correctly. by VLLM if I remember correctly. by VLLM if I remember correctly. LLM compressor. So there are LLM compressor. So there are LLM compressor. So there are tools available so you can tools available so you can tools available so you can do it yourself if you want. do it yourself if you want. do it yourself if you want. Well, the one we're currently Well, the one we're currently Well, the one we're currently using is already pre- using is already pre- using is already pre- quantized and we downloaded it from quantized and we downloaded it from quantized and we downloaded it from Hugging Face. All right. I think it Hugging Face. All right. I think it Hugging Face. All right. I think it should be available. Again, should be available. Again, should be available. Again, this takes some time, mainly this takes some time, mainly this takes some time, mainly because it fetches all the weights from because it fetches all the weights from because it fetches all the weights from VRAM and loads the new weights. So VRAM and loads the new weights. So VRAM and loads the new weights. So , it will take some time to do this , it will take some time to do this , it will take some time to do this . Does anyone have any other . Does anyone have any other . Does anyone have any other questions? Sorry?

questions? Sorry? questions? Sorry? Look at your VLLM file, your Look at your VLLM file, your Look at your VLLM file, your YAML file for your arguments. Look, YAML file for your arguments. Look, YAML file for your arguments. Look, oh, I meant it looks like they oh, I meant it looks like they oh, I meant it looks like they had two flags on the model. had two flags on the model. had two flags on the model. Oh, they did. Oh, that's a good call. Oh, they did. Oh, that's a good call. Oh, they did. Oh, that's a good call. Thank you for that. I didn't see that. Thank you for that. I didn't see that. Thank you for that. I didn't see that. All right. We'll go through this All right. We'll go through this All right. We'll go through this again. Let's run this again. again. Let's run this again. Yes, that's probably why it didn't come out Yes, that's probably why it didn't come out as quickly as it should have. as quickly as it should have. No, I can't do that. No, I can't do that. Okay, are there any other questions Okay, are there any other questions Okay, are there any other questions during this process? Here you go. during this process? Here you go. Oh, how do you choose between FP8 and INT8? Oh, how do you choose between FP8 and INT8? That's a good question. It depends, right? That's a good question. It depends, right? INT8 will have slightly less INT8 will have slightly less precision than FP8. This is basically the precision than FP8. This is basically the precision than FP8. This is basically the reason. The difference is that reason. The difference is that reason. The difference is that FP8 is floating point, so FP8 is floating point, so FP8 is floating point, so you get a little more you get a little more you get a little more precision out of it. And then INT8 is an precision out of it. And then INT8 is an precision out of it. And then INT8 is an integer. So you can use either or integer. So you can use either or integer. So you can use either or . These are all the other . These are all the other . These are all the other options available. You can choose based on your options available. You can choose based on your options available. You can choose based on your load and constraints. But yes, load and constraints. But yes, load and constraints. But yes, you can also use INT8 and it should be you can also use INT8 and it should be you can also use INT8 and it should be fine. Where is it written? With a fine. Where is it written? With a fine. Where is it written? With a good fit. Ripe stack. Oh

good fit. Ripe stack. Oh good fit. Ripe stack. Oh yes. I mean, oh, oh, that's another yes. I mean, oh, oh, that's another yes. I mean, oh, oh, that's another good decision. It also depends on the good decision. It also depends on the good decision. It also depends on the GPU. Some GPUs may not GPU. Some GPUs may not GPU. Some GPUs may not support this. So GPU support support this. So GPU support support this. So GPU support is important. This is another good decision. is important. This is another good decision. is important. This is another good decision. Some GPUs may support this, Some GPUs may support this, Some GPUs may support this, but your GPU may not accelerate it. This but your GPU may not accelerate it. This but your GPU may not accelerate it. This means you may not get the means you may not get the means you may not get the full performance you full performance you full performance you expect. For example, the expect. For example, the expect. For example, the GPUs we use are GPUs we use are GPUs we use are RTX 4080 cards. They support FP8 and RTX 4080 cards. They support FP8 and RTX 4080 cards. They support FP8 and we get full precision, full we get full precision, full we get full precision, full performance, as well as hardware performance, as well as hardware performance, as well as hardware acceleration of all calculations. acceleration of all calculations. acceleration of all calculations. But if we assume we use FP4 But if we assume we use FP4 , it will not be possible and , it will not be possible and , it will not be possible and we will not see much performance increase, on the we will not see much performance increase, on the we will not see much performance increase, on the contrary, we will see contrary, we will see contrary, we will see performance decrease instead of performance increase. This is performance decrease instead of performance increase. This is performance decrease instead of performance increase. This is another thing to another thing to another thing to remember. All right. Where is it? remember. All right. Where is it? Good, it was successfully implemented. Let's take a Good, it was successfully implemented. Let's take a quick look at this. Now quick look at this. Now quick look at this. Now we see that it actually uses the we see that it actually uses the we see that it actually uses the FP8 model. Then we'll quickly FP8 model. Then we'll quickly FP8 model. Then we'll quickly move on to the next move on to the next move on to the next load test. It will be exactly the load test. It will be exactly the load test. It will be exactly the same. We'll do it with same. We'll do it with same. We'll do it with concurrency one, four, eight concurrency one, four, eight concurrency one, four, eight and eight, 16 and 32, I think, here.

and eight, 16 and 32, I think, here. and eight, 16 and 32, I think, here. It will be exactly the same. So. And It will be exactly the same. So. And It will be exactly the same. So. And then, once it's ready, we can then, once it's ready, we can then, once it's ready, we can compare and see the compare and see the compare and see the performance differences and why it's worth performance differences and why it's worth performance differences and why it's worth quantizing. Okay, the performance is quantizing. Okay, the performance is quantizing. Okay, the performance is ready. And if we compare this before and ready. And if we compare this before and ready. And if we compare this before and after, you see, let me after, you see, let me after, you see, let me zoom in on this a little bit. You can see that even zoom in on this a little bit. You can see that even zoom in on this a little bit. You can see that even at concurrency one, you know, at concurrency one, you know, at concurrency one, you know, we go from 37.5 tokens per we go from 37.5 tokens per we go from 37.5 tokens per second to 53.4 tokens per second. second to 53.4 tokens per second. second to 53.4 tokens per second. So we see a significant increase in So we see a significant increase in So we see a significant increase in performance here. And then at peak performance here. And then at peak performance here. And then at peak throughput, we see at throughput, we see at throughput, we see at 64 concurrency, we go from 64 concurrency, we go from 64 concurrency, we go from 800 to 1140. That's about, you know, 1.442 800 to 1140. That's about, you know, 1.442 800 to 1140. That's about, you know, 1.442 times, which is a pretty big times, which is a pretty big times, which is a pretty big performance jump. But again, remember that performance jump. But again, remember that performance jump. But again, remember that the time to first token also the time to first token also the time to first token also increases significantly compared to 32 and increases significantly compared to 32 and increases significantly compared to 32 and 64. Well, but then yes, you can and 64. Well, but then yes, you can and 64. Well, but then yes, you can and will run evaluations again. Well will run evaluations again. Well will run evaluations again. Well , we can quickly look through it and there will , we can quickly look through it and there will , we can quickly look through it and there will probably be no probably be no probably be no differences. So yeah, BF16 vs differences. So yeah, BF16 vs differences. So yeah, BF16 vs FP8 again, well, there's not much of a FP8 again, well, there's not much of a FP8 again, well, there's not much of a performance hit that you performance hit that you performance hit that you should expect. In should expect. In should expect. In most cases it will most cases it will most cases it will work out of the box, but in certain work out of the box, but in certain work out of the box, but in certain use cases it may not. But for the use cases it may not. But for the use cases it may not. But for the simple evaluation cases we had simple evaluation cases we had simple evaluation cases we had here, it works fine.

here, it works fine. I think that's all for this module. I think that's all for this module. Next we'll move on to Next we'll move on to Next we'll move on to speculative decoding, which I find speculative decoding, which I find speculative decoding, which I find quite interesting. Well, let quite interesting. Well, let quite interesting. Well, let 's see. Let's start here. Okay, 's see. Let's start here. Okay, 's see. Let's start here. Okay, so how many of you have heard of so how many of you have heard of so how many of you have heard of speculative decoding before? Okay, speculative decoding before? Okay, speculative decoding before? Okay, quite a few people. That's good. For quite a few people. That's good. For quite a few people. That's good. For those who don't know what those who don't know what those who don't know what speculative decoding is, let me speculative decoding is, let me speculative decoding is, let me simplify it. Essentially, simplify it. Essentially, simplify it. Essentially, we are using another working model we are using another working model we are using another working model that will be much smaller than the that will be much smaller than the that will be much smaller than the main model we are using and main model we are using and main model we are using and will generate many tokens in will generate many tokens in will generate many tokens in advance. Then our main advance. Then our main advance. Then our main model, which is the 4 billion model, which is the 4 billion model, which is the 4 billion parameter model, will analyze these parameter model, will analyze these parameter model, will analyze these generated tokens and determine whether they generated tokens and determine whether they generated tokens and determine whether they either look good or not. In either look good or not. In either look good or not. In this diagram, you can see that the working model this diagram, you can see that the working model this diagram, you can see that the working model has generated four tokens in has generated four tokens in has generated four tokens in advance, and then the advance, and then the advance, and then the validation model, which is our model with 4 validation model, which is our model with 4 validation model, which is our model with 4 billion parameters, will say, " billion parameters, will say, " Oh, only the first and second tokens make Oh, only the first and second tokens make Oh, only the first and second tokens make sense. The other two don't." So he will sense. The other two don't." So he will sense. The other two don't." So he will reject him. And it will essentially do this reject him. And it will essentially do this reject him. And it will essentially do this every time it every time it every time it generates new tokens, so it will generates new tokens, so it will generates new tokens, so it will work in a loop. And for the purposes of work in a loop. And for the purposes of work in a loop. And for the purposes of this workshop, we will use the 0.6 this workshop, we will use the 0.6 this workshop, we will use the 0.6 billion parameter model from Cohere as the billion parameter model from Cohere as the billion parameter model from Cohere as the drafter model and see if it drafter model and see if it drafter model and see if it helps us or not. In many helps us or not. In many helps us or not. In many cases, this may also not cases, this may also not cases, this may also not help and reduce performance. And there are

help and reduce performance. And there are help and reduce performance. And there are many reasons, and many reasons, and many reasons, and we'll get to that in a moment. How are we doing with the deadline? we'll get to that in a moment. How are we doing with the deadline? we'll get to that in a moment. How are we doing with the deadline? All right. Let's just make sure All right. Let's just make sure All right. Let's just make sure everything is okay. Okay, everything is okay. Okay, everything is okay. Okay, target model. All right. Before target model. All right. Before target model. All right. Before we get into that, I also want to briefly we get into that, I also want to briefly we get into that, I also want to briefly discuss all the different ways of discuss all the different ways of speculative decoding. There are many speculative decoding. There are many different ways and different ways and different ways and new ones appear every few months. But, new ones appear every few months. But, new ones appear every few months. But, you know, the draft model is one you know, the draft model is one you know, the draft model is one way. A lot of times, a lot of the new way. A lot of times, a lot of the new way. A lot of times, a lot of the new ways of doing this are ways of doing this are ways of doing this are using a diffusion sketch model, which using a diffusion sketch model, which using a diffusion sketch model, which I think is really cool. I think is really cool. I think is really cool. I personally use this in one of my home I personally use this in one of my home I personally use this in one of my home lab machines. And this gives lab machines. And this gives lab machines. And this gives enormous efficiency. The difference between enormous efficiency. The difference between enormous efficiency. The difference between this and this is that it is a this and this is that it is a this and this is that it is a diffusion model, it will generate diffusion model, it will generate diffusion model, it will generate multiple tokens at once, instead of doing what multiple tokens at once, instead of doing what multiple tokens at once, instead of doing what we do sequentially with LLM. And we do sequentially with LLM. And we do sequentially with LLM. And then the validator model is then the validator model is then the validator model is able to verify a certain able to verify a certain able to verify a certain number of tokens from that. But But yes, there are number of tokens from that. But But yes, there are number of tokens from that. But But yes, there are many There are many different types of many There are many different types of many There are many different types of methods that you can use. You can methods that you can use. You can methods that you can use. You can use suffix decoding. It is common for a use suffix decoding. It is common for a use suffix decoding. It is common for a model to come model to come model to come with a built-in sketch model.

with a built-in sketch model. with a built-in sketch model. So these are native models that are So these are native models that are So these are native models that are trained along with the main model. trained along with the main model. trained along with the main model. So you don't need to import another So you don't need to import another So you don't need to import another model to do this. There's Medusa, model to do this. There's Medusa, model to do this. There's Medusa, Eagle, again like Diffusion, D-Flash is a Eagle, again like Diffusion, D-Flash is a Eagle, again like Diffusion, D-Flash is a big issue that went big issue that went big issue that went viral. And it works quite well. viral. And it works quite well. You were talking about a configuration that You were talking about a configuration that determines how long, for example, it takes to determines how long, for example, it takes to determines how long, for example, it takes to generate tokens before the validator generate tokens before the validator generate tokens before the validator verifies itself. You can set it up. verifies itself. You can set it up. verifies itself. You can set it up. You can set two tokens, four You can set two tokens, four You can set two tokens, four tokens and five tokens. It's important to tokens and five tokens. It's important to tokens and five tokens. It's important to set this up correctly because set this up correctly because set this up correctly because if, for example, you set it to if, for example, you set it to if, for example, you set it to four and the model validator comes in, four and the model validator comes in, four and the model validator comes in, it'll say, "OK, I'm only accepting two it'll say, "OK, I'm only accepting two ." This gives a 50% acceptance rate. And ." This gives a 50% acceptance rate. And ." This gives a 50% acceptance rate. And that's pretty bad. This will that's pretty bad. This will that's pretty bad. This will have a negative impact because not have a negative impact because not have a negative impact because not only are you running two models, so only are you running two models, so only are you running two models, so you're dedicating a computer to two models and you're dedicating a computer to two models and you're dedicating a computer to two models and then you only get a 50% then you only get a 50% then you only get a 50% acceptance rate. And this will essentially lead acceptance rate. And this will essentially lead acceptance rate. And this will essentially lead to a decrease in throughput. Uh, well, to a decrease in throughput. Uh, well, to a decrease in throughput. Uh, well, that's not what you want. Uh, that's not what you want. Uh, that's not what you want. Uh, so, this is something you'll have to tweak a little bit so, this is something you'll have to tweak a little bit so, this is something you'll have to tweak a little bit and play around with and and play around with and and play around with and see how many tokens I want to see how many tokens I want to see how many tokens I want to generate ahead of time. Uh, so generate ahead of time. Uh, so , yeah, technically you can generate , yeah, technically you can generate , yeah, technically you can generate 15 tokens, but you probably don't want to 15 tokens, but you probably don't want to 15 tokens, but you probably don't want to .

. . You probably want to do two, maybe three, maybe You probably want to do two, maybe three, maybe You probably want to do two, maybe three, maybe four. It also depends on four. It also depends on four. It also depends on what kind of um What is it called? what kind of um What is it called? what kind of um What is it called? Uh, let me come back. Where is it? Uh, let me come back. Where is it? Uh, let me come back. Where is it? What type of method do you use? What type of method do you use? What type of method do you use? Again, like uh I use D flash uh for Again, like uh I use D flash uh for Again, like uh I use D flash uh for a lot of my home labs, and a lot of my home labs, and a lot of my home labs, and uh it works pretty well if uh it works pretty well if uh it works pretty well if I set it to eight. And if I set I set it to eight. And if I set I set it to eight. And if I set it to more than eight, it to more than eight, it to more than eight, I actually get a much, uh, I actually get a much, uh, I actually get a much, uh, much lower much lower much lower acceptance rate, and that causes my acceptance rate, and that causes my acceptance rate, and that causes my performance to drop or be very performance to drop or be very performance to drop or be very unstable. Um, but Good. So, unstable. Um, but Good. So, unstable. Um, but Good. So, then um then um then um you have one more question. One you have one more question. One you have one more question. One more more more question? question? question? Hey, uh, Hey, uh, Hey, uh, sorry. sorry. Uh This is from the last module, but uh Uh This is from the last module, but uh how does dynamic compression work uh? Well how does dynamic compression work uh? Well how does dynamic compression work uh? Well , in one or two sentences, , in one or two sentences, , in one or two sentences, I think. I think. Well, the way it works is that, if Well, the way it works is that, if you understand, the different heads of the model, you understand, the different heads of the model, you understand, the different heads of the model, different because different because different because there are different layers in the LLM model. So if there are different layers in the LLM model. So if there are different layers in the LLM model. So if you look at model.json, you'll see that the you look at model.json, you'll see that the you look at model.json, you'll see that the LLM model is made up of a lot of LLM model is made up of a lot of LLM model is made up of a lot of different parts, and there's a whole different parts, and there's a whole different parts, and there's a whole process that kind of process that kind of process that kind of passes tokens from one piece to passes tokens from one piece to passes tokens from one piece to another. In many, many cases, another. In many, many cases, another. In many, many cases, when you see something dynamic, it when you see something dynamic, it when you see something dynamic, it basically says, "OK, I'll basically says, "OK, I'll basically says, "OK, I'll only quantize the heads of the notes to FP8, but only quantize the heads of the notes to FP8, but only quantize the heads of the notes to FP8, but leave everything else as is."

leave everything else as is." leave everything else as is." I get it. So, I get it. So, I get it. So, that's the reasoning. And if that's the reasoning. And if that's the reasoning. And if you remember from the previous modules, you remember from the previous modules, you remember from the previous modules, you saw that the FP8 version isn't you saw that the FP8 version isn't you saw that the FP8 version isn't exactly half the size. It's not exactly half the size. It's not exactly half the size. It's not 4GB, it's about 5.1GB. That's because 4GB, it's about 5.1GB. That's because 4GB, it's about 5.1GB. That's because not every part of the model has been not every part of the model has been not every part of the model has been quantized to FP8. That's the quantized to FP8. That's the quantized to FP8. That's the dynamic part. dynamic part. dynamic part. I get it. Does the same math I get it. Does the same math I get it. Does the same math apply to all models, or does it apply to all models, or does it apply to all models, or does it depend on the depend on the depend on the model? Every model has a different model? Every model has a different model? Every model has a different architecture, so architecturally architecture, so architecturally architecture, so architecturally speaking, it depends, you know, speaking, it depends, you know, speaking, it depends, you know, which parts of the model you're quantizing. which parts of the model you're quantizing. which parts of the model you're quantizing. There are people who quantize, you know, There are people who quantize, you know, There are people who quantize, you know, some parts of the model in quads, some parts of the model in quads, some parts of the model in quads, but leave other parts for FP8, but leave other parts for FP8, but leave other parts for FP8, and some parts for BF16. So and some parts for BF16. So and some parts for BF16. So thank you. thank you. thank you. Yes. Okay, we'll go through that. Yes. Okay, we'll go through that. Yes. Okay, we'll go through that. Again, just confirming that Again, just confirming that Again, just confirming that our baseline is good. We're using the our baseline is good. We're using the our baseline is good. We're using the FP8 model here. And then FP8 model here. And then FP8 model here. And then we'll quickly run another test. we'll quickly run another test. we'll quickly run another test. load to get another load to get another load to get another baseline of what this will baseline of what this will baseline of what this will look like before we implement and look like before we implement and look like before we implement and add speculative decoding to this add speculative decoding to this add speculative decoding to this particular implementation in vLLM. And then, particular implementation in vLLM. And then, particular implementation in vLLM. And then, while this is running, I'll make sure while this is running, I'll make sure while this is running, I'll make sure I can copy it correctly. No, I can copy it correctly. No, I can copy it correctly. No, not this one. Oh, never mind. You can not this one. Oh, never mind. You can

not this one. Oh, never mind. You can just, uh, here, uh, we set the just, uh, here, uh, we set the just, uh, here, uh, we set the speculative configuration. This, uh, this speculative configuration. This, uh, this speculative configuration. This, uh, this is a specific flag that you can is a specific flag that you can is a specific flag that you can pass in vLLM to set the pass in vLLM to set the pass in vLLM to set the working model. We set it to a model with, uh, working model. We set it to a model with, uh, working model. We set it to a model with, uh, 0.6 billion parameters. Then 0.6 billion parameters. Then 0.6 billion parameters. Then we also set how many tokens we also set how many tokens we also set how many tokens we want to generate as working tokens we want to generate as working tokens . In this particular scenario, . In this particular scenario, . In this particular scenario, we set it to four. Again, this we set it to four. Again, this we set it to four. Again, this is something that needs a lot of testing. is something that needs a lot of testing. is something that needs a lot of testing. Uh, you have to try it out, do I want to Uh, you have to try it out, do I want to Uh, you have to try it out, do I want to set it to four, do I want to set it to four, do I want to set it to four, do I want to set it to two, do I want to set set it to two, do I want to set set it to two, do I want to set it higher? It all depends on the it higher? It all depends on the it higher? It all depends on the acceptance rate and, uh, how much acceptance rate and, uh, how much acceptance rate and, uh, how much performance you get from it. Uh, and performance you get from it. Uh, and performance you get from it. Uh, and then just like you set this flag, then just like you set this flag, then just like you set this flag, we'll deploy it again. Uh, we're back we'll deploy it again. Uh, we're back we'll deploy it again. Uh, we're back here. We have our baseline. And here. We have our baseline. And here. We have our baseline. And then let me make sure it then let me make sure it then let me make sure it 's good. Okay. Everything looks 's good. Okay. Everything looks 's good. Okay. Everything looks good, and then we'll go ahead good, and then we'll go ahead good, and then we'll go ahead and, uh, apply this and deploy. And this and, uh, apply this and deploy. And this and, uh, apply this and deploy. And this will probably take a few more will probably take a few more will probably take a few more minutes, so does anyone have any minutes, so does anyone have any minutes, so does anyone have any questions? Here you go. Oh, one second.

questions? Here you go. Oh, one second. I noticed you set the I noticed you set the GPU memory utilization to 0.7, which GPU memory utilization to 0.7, which GPU memory utilization to 0.7, which leaves a lot, er, actually leaves a lot, er, actually leaves a lot, er, actually leaves 30% of the VRAM leaves 30% of the VRAM leaves 30% of the VRAM unused. Is there any unused. Is there any unused. Is there any specific logic to this choice? specific logic to this choice? specific logic to this choice? Um, in this particular scenario, er, there Um, in this particular scenario, er, there Um, in this particular scenario, er, there could be a number of reasons. Um, sometimes could be a number of reasons. Um, sometimes , you know, you might have multiple deployments , you know, you might have multiple deployments , you know, you might have multiple deployments on the same GPU and you want to make sure on the same GPU and you want to make sure you have enough you have enough you have enough RAM for something else, VRAM for RAM for something else, VRAM for RAM for something else, VRAM for something else, and then you keep something else, and then you keep something else, and then you keep enough VRAM for that. But in enough VRAM for that. But in enough VRAM for that. But in this particular scenario, you do this particular scenario, you do this particular scenario, you do n't really. We'll turn this up in the n't really. We'll turn this up in the n't really. We'll turn this up in the next module, actually. Um, next module, actually. Um, next module, actually. Um, this is one of those knobs, right this is one of those knobs, right ? You have to make sure you're setting ? You have to make sure you're setting ? You have to make sure you're setting the utilization correctly. If the utilization correctly. If the utilization correctly. If you set it too low, performance you set it too low, performance you set it too low, performance will suffer. If If you set it will suffer. If If you set it will suffer. If If you set it too high, uh, you'll kill your node, too high, uh, you'll kill your node, too high, uh, you'll kill your node, and that's not good.

and that's not good. So if you go to So if you go to one, one, one, is that up to date? is that up to date? Yes, PR. You Yes, PR. You don't have a number. don't have a number. don't have a number. Okay, it still shows up. It takes Okay, it still shows up. It takes Okay, it still shows up. It takes a little longer. Next. a little longer. Next. We see parallelization from 32 to 64, sorry, one second. sorry, one second. Um, the jump we see is going Um, the jump we see is going Um, the jump we see is going from 60 to 32 from 60 to 32 from 60 to 32 to 64 parallelization, that's because the to 64 parallelization, that's because the to 64 parallelization, that's because the key value cache is key value cache is key value cache is simply exhausting, or I simply exhausting, or I simply exhausting, or I mean, mean, mean, yes. Basically, the yes. Basically, the yes. Basically, the key value cache is key value cache is key value cache is exhausting and it's kind of exhausting and it's kind of exhausting and it's kind of reached its maximum limit and reached its maximum limit and reached its maximum limit and can't go beyond that. Um, there are a lot of can't go beyond that. Um, there are a lot of can't go beyond that. Um, there are a lot of reasons. There's that part, and then there reasons. There's that part, and then there reasons. There's that part, and then there 's also, uh, continuous 's also, uh, continuous 's also, uh, continuous batching. And many times what batching. And many times what batching. And many times what happens is if you don't set happens is if you don't set happens is if you don't set the configuration to a level where the configuration to a level where the configuration to a level where continuous batching is not continuous batching is not continuous batching is not optimal, it can also optimal, it can also optimal, it can also cause you to see high cause you to see high cause you to see high tokens per second, uh, tokens per second, uh, tokens per second, uh, you might see high latency there as well.

you might see high latency there as well. you might see high latency there as well. Um, but yes, in this Um, but yes, in this Um, but yes, in this scenario, We're essentially hitting something scenario, We're essentially hitting something scenario, We're essentially hitting something of a ceiling, um, for us, in of a ceiling, um, for us, in of a ceiling, um, for us, in terms of the amount of VRAM terms of the amount of VRAM terms of the amount of VRAM we can use there. That's we can use there. That's we can use there. That's where we see it. But in the next modules, where we see it. But in the next modules, where we see it. But in the next modules, we'll see how we can improve this we'll see how we can improve this we'll see how we can improve this scenario and hopefully scenario and hopefully scenario and hopefully improve it. Um, okay. Okay, improve it. Um, okay. Okay, improve it. Um, okay. Okay, we've implemented it. So now we have we've implemented it. So now we have speculative decoding configured correctly. So what we're going to do is speculative decoding configured correctly. So what we're going to do is run a quick load test again run a quick load test again run a quick load test again . It's the exact same . It's the exact same . It's the exact same test we ran here. test we ran here. test we ran here. You know, remember, we're getting 58.2 You know, remember, we're getting 58.2 You know, remember, we're getting 58.2 tokens per second for tokens per second for tokens per second for concurrency 1, and then a 42 concurrency 1, and then a 42 concurrency 1, and then a 42 millisecond latency. So that's millisecond latency. So that's millisecond latency. So that's our baseline before we do our baseline before we do our baseline before we do speculative decoding. But what I speculative decoding. But what I speculative decoding. But what I want to show you here is not a want to show you here is not a want to show you here is not a happy path that will happy path that will happy path that will automatically deliver incredible automatically deliver incredible automatically deliver incredible performance improvements. In fact, performance improvements. In fact, performance improvements. In fact, you'll see a drastic performance drop you'll see a drastic performance drop . You're seeing 16%16 tokens per second, . You're seeing 16%16 tokens per second, . You're seeing 16%16 tokens per second, while the baseline was while the baseline was while the baseline was 58. And even the time to first token has 58. And even the time to first token has 58. And even the time to first token has increased a bit. And, you know, increased a bit. And, you know, increased a bit. And, you know, we want to figure out why that we want to figure out why that we want to figure out why that happened. What's going on? So there's a big difference happened. What's going on? So there's a big difference happened. What's going on? So there's a big difference and a big performance hit. Even and a big performance hit. Even and a big performance hit. Even if you look at if you look at if you look at 16 concurrency, we're going from 700 tokens per 16 concurrency, we're going from 700 tokens per 16 concurrency, we're going from 700 tokens per second to 240 tokens per second.

second to 240 tokens per second. second to 240 tokens per second. That's a significant performance hit. There are a That's a significant performance hit. There are a lot of different reasons for this. One of them lot of different reasons for this. One of them is, as I said before, a is, as I said before, a is, as I said before, a low acceptance rate. Well low acceptance rate. Well low acceptance rate. Well , if the acceptance rate is , if the acceptance rate is , if the acceptance rate is very low and the validation model very low and the validation model very low and the validation model simply rejects the tokens simply rejects the tokens simply rejects the tokens generated by the working model, generated by the working model, generated by the working model, you're wasting a lot of processing power you're wasting a lot of processing power you're wasting a lot of processing power generating those working tokens, generating those working tokens, generating those working tokens, and then the validator comes in and and then the validator comes in and and then the validator comes in and says, "Hey, I don't want to says, "Hey, I don't want to says, "Hey, I don't want to take on this job." This is a bad job. take on this job." This is a bad job. take on this job." This is a bad job. "Well, that's one of the reasons. "Well, that's one of the reasons. "Well, that's one of the reasons. It's possible that It's possible that It's possible that we've set the speculative token values ​​too high we've set the speculative token values ​​too high we've set the speculative token values ​​too high , meaning , meaning , meaning that in our current setup, that in our current setup, that in our current setup, we've set them to four. Maybe we've set them to four. Maybe we've set them to four. Maybe we should lower them and see we should lower them and see we should lower them and see if that actually helps improve the if that actually helps improve the if that actually helps improve the acceptance criteria. Or the acceptance criteria. Or the acceptance criteria. Or the acceptance rate. A lot of things acceptance rate. A lot of things acceptance rate. A lot of things are also related to the are also related to the are also related to the scheduling in VLLM, depending on scheduling in VLLM, depending on scheduling in VLLM, depending on how it schedules these requests. how it schedules these requests. how it schedules these requests. That can degrade performance. So That can degrade performance. So That can degrade performance. So , it also depends on , it also depends on , it also depends on compatibility and things like that compatibility and things like that . And so, again, the same problem with GPUs. . And so, again, the same problem with GPUs. . And so, again, the same problem with GPUs. You're using the same GPU to do You're using the same GPU to do You're using the same GPU to do the job. So, it can cause the job. So, it can cause the job. So, it can cause resources to not be resources to not be resources to not be utilized correctly. In many cases, that utilized correctly. In many cases, that utilized correctly. In many cases, that usually works. But in our usually works. But in our usually works. But in our scenario, it didn't work. The scenario, it didn't work. The scenario, it didn't work. The main reason is that this main reason is that this main reason is that this particular working model isn't particular working model isn't particular working model isn't trained correctly for this trained correctly for this trained correctly for this particular model. That's why we're seeing a particular model. That's why we're seeing a particular model. That's why we're seeing a very, very low very, very low very, very low acceptance rate. When you do this for your own

acceptance rate. When you do this for your own acceptance rate. When you do this for your own models and your own implementations, you need to models and your own implementations, you need to models and your own implementations, you need to make sure that the working model... The make sure that the working model... The GPU you're using GPU you're using GPU you're using is compatible with the is compatible with the is compatible with the model you want to use it for. model you want to use it for. model you want to use it for. If you don't do that, you'll run into If you don't do that, you'll run into If you don't do that, you'll run into the scenario I'm presenting here, the scenario I'm presenting here, the scenario I'm presenting here, where performance will drop and not much will change." It's where performance will drop and not much will change." It's where performance will drop and not much will change." It's doable. I can say doable. I can say doable. I can say that from a resource perspective, this that from a resource perspective, this that from a resource perspective, this GPU has enough GPU has enough GPU has enough headroom to handle all of this, headroom to handle all of this, headroom to handle all of this, but the reason for the performance drop is but the reason for the performance drop is but the reason for the performance drop is that the models aren't being trained correctly. that the models aren't being trained correctly. that the models aren't being trained correctly. And that's something to And that's something to And that's something to keep in mind and something to test as well keep in mind and something to test as well keep in mind and something to test as well . And you need to make sure . And you need to make sure . And you need to make sure that the speculative tokens are set that the speculative tokens are set that the speculative tokens are set to the right number. Well, in your to the right number. Well, in your to the right number. Well, in your free time, I strongly encourage free time, I strongly encourage free time, I strongly encourage everyone to play around with that number, everyone to play around with that number, everyone to play around with that number, so change it from four to two to so change it from four to two to so change it from four to two to six to one, and see what six to one, and see what six to one, and see what performance gain you see, what performance gain you see, what performance gain you see, what performance loss you see, and why performance loss you see, and why performance loss you see, and why . But yeah, that's it for . But yeah, that's it for . But yeah, that's it for our speculative decoding. It our speculative decoding. It our speculative decoding. It won't really help us in this won't really help us in this won't really help us in this scenario. So scenario. So scenario. So I'll just comment on that. And then we'll I'll just comment on that. And then we'll I'll just comment on that. And then we'll reimplement it. So we're back to square one reimplement it. So we're back to square one reimplement it. So we're back to square one .

. Hey, Omar, you have one question. Hey, Omar, you have one question. One question? One question? One question? Hey, um, I know there's probably Hey, um, I know there's probably Hey, um, I know there's probably some nuance some nuance some nuance depending on the model architecture, depending on the model architecture, depending on the model architecture, but overall, from what you've seen, is there a but overall, from what you've seen, is there a but overall, from what you've seen, is there a significant drop in significant drop in significant drop in acceptance rate if acceptance rate if acceptance rate if you use essentially mixed precision, you use essentially mixed precision, you use essentially mixed precision, for example, you run master model for example, you run master model for example, you run master model in FP8, and then quantize the in FP8, and then quantize the in FP8, and then quantize the working model to Q2? If I have really working model to Q2? If I have really working model to Q2? If I have really limited memory, is limited memory, is limited memory, is this often worth doing? Um, or this often worth doing? Um, or this often worth doing? Um, or do you know what you've generally seen? Yes, does it do you know what you've generally seen? Yes, does it do you know what you've generally seen? Yes, does it have a have a have a bad... bad... bad... That's a good question. Um, often... uh, That's a good question. Um, often... uh, it doesn't matter that much. Um, in it doesn't matter that much. Um, in most cases, you want most cases, you want most cases, you want to keep the working model and the to keep the working model and the to keep the working model and the master model with the same precision. But master model with the same precision. But master model with the same precision. But let's say the working model is... let's say the working model is... BF16, and then the master model is BF16, and then the master model is BF16, and then the master model is in FP8. Mismatching can result in a in FP8. Mismatching can result in a in FP8. Mismatching can result in a very low acceptance rate. very low acceptance rate. very low acceptance rate. So, then the most important So, then the most important So, then the most important part is to properly part is to properly part is to properly train the working model so that it train the working model so that it train the working model so that it can generate the appropriate tokens can generate the appropriate tokens . In our specific scenario, the . In our specific scenario, the . In our specific scenario, the working model wasn't working model wasn't working model wasn't trained correctly. It was also trained correctly. It was also trained correctly. It was also quantized to FP8.

quantized to FP8. quantized to FP8. There were probably some issues with this There were probably some issues with this There were probably some issues with this recipe during training of this recipe during training of this recipe during training of this model, which resulted in a significant model, which resulted in a significant model, which resulted in a significant performance drop. Often, when a lab performance drop. Often, when a lab performance drop. Often, when a lab publishes a model, they also publish the publishes a model, they also publish the publishes a model, they also publish the working model. Therefore, it's worth using working model. Therefore, it's worth using working model. Therefore, it's worth using these working models that are these working models that are these working models that are included with the model. There are included with the model. There are included with the model. There are also external labs also external labs also external labs that create their own working models. that create their own working models. that create their own working models. You can So, try a lot of You can So, try a lot of You can So, try a lot of different options available and different options available and different options available and see which one works. In see which one works. In see which one works. In some cases, depending some cases, depending some cases, depending on the use case, let's say on the use case, let's say on the use case, let's say you're using it for coding, the working model you're using it for coding, the working model you're using it for coding, the working model might work perfectly, but if might work perfectly, but if might work perfectly, but if you change it to something like English, you change it to something like English, you change it to something like English, writing, very specific writing, very specific writing, very specific academic writing, it won't necessarily academic writing, it won't necessarily academic writing, it won't necessarily work well. And you'll see a drop in work well. And you'll see a drop in work well. And you'll see a drop in acceptance rate. So, it acceptance rate. So, it acceptance rate. So, it also depends a lot on also depends a lot on also depends a lot on your use case. So it your use case. So it your use case. So it 's not universal. 's not universal. 's not universal. I understand. Great. Makes sense. Thanks. I understand. Great. Makes sense. Thanks. I understand. Great. Makes sense. Thanks. Yes. Okay. Yes. Okay. Let's move on to the Let's move on to the inference side. But before we go any inference side. But before we go any inference side. But before we go any further, any other questions?

further, any other questions? Oh. Oh. Yes, I can put YAML there. Go Yes, I can put YAML there. Go Yes, I can put YAML there. Go ahead. ahead. The fact that we have speculative decoding is probably because it's probably because it's probably because it's faster for the model to verify than faster for the model to verify than faster for the model to verify than decode. decode. Yes. Yes. Can you provide some Can you provide some Can you provide some mental model of what the model is doing " mental model of what the model is doing " under the hood" to verify so that it's under the hood" to verify so that it's under the hood" to verify so that it's faster than decoding? faster than decoding? faster than decoding? You know, it can't be decoding and You know, it can't be decoding and You know, it can't be decoding and then comparing it to what the then comparing it to what the then comparing it to what the working model generates. So working model generates. So working model generates. So what exactly does it do to make it what exactly does it do to make it what exactly does it do to make it faster? faster? faster? Yes. So basically, when a typical LLM Yes. So basically, when a typical LLM , you know, decodes, you have a , you know, decodes, you have a , you know, decodes, you have a vocabulary set. It's a huge array of lots of vocabulary set. It's a huge array of lots of vocabulary set. It's a huge array of lots of numbers that represents the, um, numbers that represents the, um, numbers that represents the, um, trust score for each of these items trust score for each of these items , right? So instead of, um, instead of , right? So instead of, um, instead of , right? So instead of, um, instead of executing this one token at a time, executing this one token at a time, executing this one token at a time, when you're doing speculative decoding, when you're doing speculative decoding, when you're doing speculative decoding, you're running it for multiple things, you're running it for multiple things, you're running it for multiple things, you're running the model for you're running the model for you're running the model for parallel things. So, in parallel things. So, in parallel things. So, in continuous batch processing, continuous batch processing, continuous batch processing, you run many different requests you run many different requests you run many different requests in parallel. You do the same thing, you do in parallel. You do the same thing, you do in parallel. You do the same thing, you do the calculations for everything, and you do it the calculations for everything, and you do it the calculations for everything, and you do it for many different for many different for many different requests simultaneously. In the same way, you do the same thing requests simultaneously. In the same way, you do the same thing requests simultaneously. In the same way, you do the same thing for speculative decoding.

for speculative decoding. for speculative decoding. You check the trust score for You check the trust score for You check the trust score for speculated tokens and make sure it's speculated tokens and make sure it's high trust, high trust, high trust, so we'll try that. This one is, you know, so we'll try that. This one is, you know, so we'll try that. This one is, you know, 0.95, and this one is about 0.9. And then you 0.95, and this one is about 0.9. And then you 0.95, and this one is about 0.9. And then you accept it. And then the rest, accept it. And then the rest, accept it. And then the rest, you know, you throw away. It's more or less like this you know, you throw away. It's more or less like this you know, you throw away. It's more or less like this in the mental model... in the mental model... going through going through going through the layers in parallel and making sure that each of the the layers in parallel and making sure that each of the the layers in parallel and making sure that each of the tokens generated by the tokens generated by the tokens generated by the draft model has a high level of credibility. draft model has a high level of credibility. Yes. I hope this answers Yes. I hope this answers your question. Oh, one more question your question. Oh, one more question ? Yes, yes. That's a good call. Hmm, ? Yes, yes. That's a good call. Hmm, ? Yes, yes. That's a good call. Hmm, I actually left a few I actually left a few I actually left a few links here. Where did I leave them? links here. Where did I leave them? links here. Where did I leave them? Interpretation of results. Yes, there are many Interpretation of results. Yes, there are many Interpretation of results. Yes, there are many different studies on different studies on different studies on speculative decoding. speculative decoding. speculative decoding. Overall, if you do it right Overall, if you do it right , you will see a significant increase in performance. , you will see a significant increase in performance. , you will see a significant increase in performance. VLLM actually did a comprehensive VLLM actually did a comprehensive VLLM actually did a comprehensive report on this and saw report on this and saw report on this and saw about a 1.5x increase in about a 1.5x increase in about a 1.5x increase in token generation if you set it up token generation if you set it up token generation if you set it up correctly with the right model, with the correctly with the right model, with the correctly with the right model, with the right drafter, and with the right right drafter, and with the right right drafter, and with the right settings. Yes, yes, it settings. Yes, yes, it settings. Yes, yes, it works. It's just about works. It's just about works. It's just about doing it right, tuning it correctly, doing it right, tuning it correctly, doing it right, tuning it correctly, and making sure you have the right and making sure you have the right and making sure you have the right things. But yes, from my personal things. But yes, from my personal things. But yes, from my personal experience, experience, experience, I used Qwen 3.6 Model I used Qwen 3.6 Model I used Qwen 3.6 Model MoE 35 billion in my home lab. In addition,

MoE 35 billion in my home lab. In addition, MoE 35 billion in my home lab. In addition, I used the Deep Flash drafter model. And I used the Deep Flash drafter model. And I used the Deep Flash drafter model. And I get about a two to three times I get about a two to three times I get about a two to three times improvement in token generation performance improvement in token generation performance improvement in token generation performance , depending on the , depending on the , depending on the use case. So it works. It's just a matter of making the use case. So it works. It's just a matter of making the use case. So it works. It's just a matter of making the right models right models right models work together. work together. How to choose the size of the model for? Is How to choose the size of the model for? Is there a sweet spot there a sweet spot there a sweet spot for a draftee? for a draftee? for a draftee? For the drafter model? For the drafter model? Basically, you want to find the smallest Basically, you want to find the smallest model you can run model you can run model you can run that doesn't use as much GPU power that doesn't use as much GPU power that doesn't use as much GPU power because... The master model is the model because... The master model is the model because... The master model is the model that has all the knowledge. This is the model that has all the knowledge. This is the model that has all the knowledge. This is the model you really care about and that you really care about and that you really care about and that has everything you need. But has everything you need. But has everything you need. But you don't want the draft model to take up you don't want the draft model to take up you don't want the draft model to take up a lot of resources from the main model. a lot of resources from the main model. a lot of resources from the main model. So you want it to be small, but not So you want it to be small, but not So you want it to be small, but not so small that it's stupid, but so small that it's stupid, but so small that it's stupid, but big enough that it actually big enough that it actually big enough that it actually gives you... gives you... gives you... Have you experimented with this to Have you experimented with this to Have you experimented with this to see those acceptance rates? see those acceptance rates? Model 3B may be Model 3B may be too big. The reason for this is… A lot of the too big. The reason for this is… A lot of the too big. The reason for this is… A lot of the MoE models I was talking about are MoE models I was talking about are MoE models I was talking about are 3.6 billion, uh, 35 billion quantities 3.6 billion, uh, 35 billion quantities . It's a mixed model, and each expert . It's a mixed model, and each expert . It's a mixed model, and each expert has about 3 billion parameters.

has about 3 billion parameters. has about 3 billion parameters. So experts can be around and So experts can be around and So experts can be around and get good performance out of it. But get good performance out of it. But get good performance out of it. But for a drafting model, for a drafting model, for a drafting model, you probably want to have less than a you probably want to have less than a you probably want to have less than a billion parameters to make sure billion parameters to make sure billion parameters to make sure that this also depends on the that this also depends on the that this also depends on the master model you're using, right? For example master model you're using, right? For example , if you're using... If your , if you're using... If your , if you're using... If your master model has about 400 billion master model has about 400 billion master model has about 400 billion parameters, you can probably parameters, you can probably parameters, you can probably afford to use a larger afford to use a larger afford to use a larger drafting model to drafting model to drafting model to make sure you get a higher make sure you get a higher make sure you get a higher acceptance rate. So, basically acceptance rate. So, basically acceptance rate. So, basically you have to find the right you have to find the right you have to find the right balance. Yes. Great. Do we have balance. Yes. Great. Do we have balance. Yes. Great. Do we have any other questions? One more thing? any other questions? One more thing? Before you go? Before you go? Yes, I just had a question. How Yes, I just had a question. How Yes, I just had a question. How is a drafting model trained? Is is a drafting model trained? Is is a drafting model trained? Is it distillation or something else? it distillation or something else? it distillation or something else? It's basically distillation. So you take a It's basically distillation. So you take a It's basically distillation. So you take a big model and, you know, distill big model and, you know, distill big model and, you know, distill all of its capabilities, create a all of its capabilities, create a all of its capabilities, create a dataset, and then…train a dataset, and then…train a dataset, and then…train a working model with that. And working model with that. And working model with that. And actually, there are often different actually, there are often different actually, there are often different techniques to do this. Another technique techniques to do this. Another technique you will see is to you will see is to you will see is to train a working model in parallel train a working model in parallel train a working model in parallel with the main model. This way is with the main model. This way is with the main model. This way is another way. Many labs another way. Many labs another way. Many labs provide a working model in the same provide a working model in the same provide a working model in the same package. So you don't have to download the package. So you don't have to download the package. So you don't have to download the second model and run it.

second model and run it. second model and run it. There are a few different ways, but There are a few different ways, but There are a few different ways, but probably the best way is to probably the best way is to probably the best way is to train alongside it, this train alongside it, this train alongside it, this will avoid a lot of problems and your will avoid a lot of problems and your will avoid a lot of problems and your acceptance rate will remain acceptance rate will remain acceptance rate will remain high. Great. Let's move on to high. Great. Let's move on to high. Great. Let's move on to module seven. Here I want to cover module seven. Here I want to cover module seven. Here I want to cover a few different flags to look out for a few different flags to look out for a few different flags to look out for when deploying when deploying when deploying with vllm, and what and how you should with vllm, and what and how you should with vllm, and what and how you should do to keep saturation as do to keep saturation as do to keep saturation as high as possible. Saturation means high as possible. Saturation means high as possible. Saturation means we want the GPU to work as we want the GPU to work as we want the GPU to work as hard as possible for as long as possible. And that's hard as possible for as long as possible. And that's hard as possible for as long as possible. And that's why we want to get the most why we want to get the most why we want to get the most for our money. If you are renting for our money. If you are renting for our money. If you are renting a GPU or buying a GPU, you want to a GPU or buying a GPU, you want to a GPU or buying a GPU, you want to run it. As much as possible. So oh. run it. As much as possible. So oh. run it. As much as possible. So oh. We will stop here. We will stop here. One of the concepts I want to One of the concepts I want to talk about, if you remember, talk about, if you remember, talk about, if you remember, earlier we were looking at earlier we were looking at earlier we were looking at 34, 32, and 64 concurrency, 34, 32, and 64 concurrency, 34, 32, and 64 concurrency, we saw a dramatic increase in we saw a dramatic increase in we saw a dramatic increase in latency. The reason for this is that latency. The reason for this is that latency. The reason for this is that bandwidth will flatten out at some point bandwidth will flatten out at some point bandwidth will flatten out at some point . But the latency can . But the latency can . But the latency can keep increasing if we keep keep increasing if we keep keep increasing if we keep increasing concurrency. So increasing concurrency. So increasing concurrency. So basically you want to find the sweet basically you want to find the sweet basically you want to find the sweet spot, and that's something that vllm allows spot, and that's something that vllm allows spot, and that's something that vllm allows you to kind of tweak and you to kind of tweak and you to kind of tweak and make sure that you find that make sure that you find that make sure that you find that sweet spot, and once you sweet spot, and once you sweet spot, and once you find it, you can set the settings for it find it, you can set the settings for it find it, you can set the settings for it . Well, but . Well, but . Well, but basically you want to find the inner basically you want to find the inner basically you want to find the inner point, not the intersection, but the point just point, not the intersection, but the point just point, not the intersection, but the point just before where the throughput

before where the throughput before where the throughput reaches a flat state but the latency reaches a flat state but the latency reaches a flat state but the latency is still acceptable enough. is still acceptable enough. is still acceptable enough. This is roughly the point This is roughly the point This is roughly the point you want to get to. Well, you want to get to. Well, you want to get to. Well, let's get setup. And then let's let's get setup. And then let's let's get setup. And then let's make sure all our make sure all our make sure all our settings are still good. The settings are still good. The settings are still good. The service model is running. Well, service model is running. Well, service model is running. Well, let me…Check carefully if it let me…Check carefully if it let me…Check carefully if it works. It should. Yes. And then works. It should. Yes. And then works. It should. Yes. And then we'll get another one again. We'll run we'll get another one again. We'll run we'll get another one again. We'll run another load test to another load test to another load test to see the data and use it to see the data and use it to see the data and use it to determine where that point is. determine where that point is. We call this the knee where things We call this the knee where things flatten out, but also flatten out, but also concurrency based concurrency, time to first concurrency based concurrency, time to first token is acceptable. When that token is acceptable. When that token is acceptable. When that happens, oh, I think it should happens, oh, I think it should happens, oh, I think it should be fine. Da da da da da da So be fine. Da da da da da da So be fine. Da da da da da da So you know, throughput slowly increases you know, throughput slowly increases you know, throughput slowly increases as concurrency increases. This as concurrency increases. This as concurrency increases. This is acceptable. Um, and then you is acceptable. Um, and then you is acceptable. Um, and then you also see an increase in latency. This also see an increase in latency. This also see an increase in latency. This is also acceptable, but does not increase is also acceptable, but does not increase is also acceptable, but does not increase drastically. Uh, but we drastically. Uh, but we drastically. Uh, but we should see big jumps soon should see big jumps soon should see big jumps soon with more concurrency here.

with more concurrency here. But now that it works, does anyone have But now that it works, does anyone have any other questions? Here you go any other questions? Here you go . Oh, there. . Oh, there. . Oh, there. So my question was about the So my question was about the So my question was about the previous speculative decoding previous speculative decoding . Um, I just wanted to know, . Um, I just wanted to know, . Um, I just wanted to know, based on your experience, uh, based on your experience, uh, based on your experience, uh, with this kind of drafting model, uh, with this kind of drafting model, uh, with this kind of drafting model, uh, what kind of, in what what kind of, in what what kind of, in what use cases have you seen it use cases have you seen it use cases have you seen it actually work better? For example, in actually work better? For example, in actually work better? For example, in agent loops, with agent loops, with agent loops, with tool calls, or with coding. What tool calls, or with coding. What tool calls, or with coding. What kind of use cases? kind of use cases? kind of use cases? Agent loops are really good at Agent loops are really good at Agent loops are really good at calling tools, they're really calling tools, they're really calling tools, they're really good at using the browser, that good at using the browser, that good at using the browser, that 's one specific use case where 's one specific use case where 's one specific use case where I use them. I want to I use them. I want to I use them. I want to automate a lot of automate a lot of automate a lot of browser tasks because still, you know, browser tasks because still, you know, browser tasks because still, you know, you don't necessarily have an API for everything. you don't necessarily have an API for everything. And you want to fill that gap, and yes, the And you want to fill that gap, and yes, the working model helps generate more working model helps generate more working model helps generate more tokens faster. So you see tokens faster. So you see tokens faster. So you see your browser agent just your browser agent just your browser agent just works and does things faster. works and does things faster. works and does things faster. So in many ways it helps. So in many ways it helps. So in many ways it helps. I would say that when I would say that when I would say that when using a browser using a browser using a browser I probably see about a 1.5x I probably see about a 1.5x I probably see about a 1.5x increase. However, in the case of coding, increase. However, in the case of coding, increase. However, in the case of coding, I see it closer to 3 times. So I see it closer to 3 times. So I see it closer to 3 times. So depending on your use case, depending on your use case, depending on your use case, you might see more or less and you might see more or less and you might see more or less and things like that.

things like that. things like that. And the continuation of this. Are there any And the continuation of this. Are there any And the continuation of this. Are there any benchmarks like "trust benchmarks like "trust benchmarks like "trust me bro", benchmarks me bro", benchmarks me bro", benchmarks like ship coding? like ship coding? like ship coding? There are definitely There are definitely There are definitely brand comparison tests out there. Personally, I'm a brand comparison tests out there. Personally, I'm a brand comparison tests out there. Personally, I'm a bit skeptical about bit skeptical about bit skeptical about benchmarks. Every time benchmarks. Every time benchmarks. Every time I see them I never get the I see them I never get the I see them I never get the same results. I don't know if people same results. I don't know if people same results. I don't know if people agree with that. So yeah, in agree with that. So yeah, in agree with that. So yeah, in most cases, you know, you want to most cases, you know, you want to most cases, you know, you want to do your own testing and do your own testing and do your own testing and see what works because it very much see what works because it very much see what works because it very much depends on your use case depends on your use case because the adoption rate because the adoption rate because the adoption rate is going to vary depending is going to vary depending is going to vary depending on your use case and what on your use case and what on your use case and what you're using it for. you're using it for. you're using it for. Thank you. Thank you. So, do you have any more questions? So, do you have any more questions? Microphone check. Microphone check. Microphone check. Microphone check. How's it going? Microphone check. How's it going? So, large models like Codex So, large models like Codex or Claude, do they use this type of or Claude, do they use this type of or Claude, do they use this type of decoding? Have they published decoding? Have they published decoding? Have they published information about using these patterns, information about using these patterns, information about using these patterns, or is this more typical for or is this more typical for or is this more typical for open source software if open source software if open source software if you're training your own stuff or you're training your own stuff or you're training your own stuff or running inference? I running inference? I don't know if they do it or not. Well don't know if they do it or not. Well , I imagine so. Sort of like , I imagine so. Sort of like , I imagine so. Sort of like free performance, but well free performance, but well free performance, but well , for open source software, , for open source software, , for open source software, yes. This is especially true for yes. This is especially true for yes. This is especially true for open source software. All the open source software. All the open source software. All the big open source models big open source models big open source models you want to use, all the Quen models, you want to use, all the Quen models, you want to use, all the Quen models, GLM models, Kimmy models, they all GLM models, Kimmy models, they all GLM models, Kimmy models, they all have speculative decoding. The have speculative decoding. The have speculative decoding. The draft models draft models draft models you want to use are available. Is you want to use are available. Is you want to use are available. Is there a main draft model there a main draft model there a main draft model they use, like they use, like they use, like Deep Seeking or GLM?

Deep Seeking or GLM? Deep Seeking released something new yesterday, or the day Deep Seeking released something new yesterday, or the day before yesterday, called before yesterday, called before yesterday, called D-Spark. This is their own technique for D-Spark. This is their own technique for D-Spark. This is their own technique for speculative decoding in a more speculative decoding in a more speculative decoding in a more dynamic way. Basically, dynamic way. Basically, dynamic way. Basically, they're figuring out how…So here's they're figuring out how…So here's they're figuring out how…So here's our static configuration, right? our static configuration, right? our static configuration, right? As we say, we will generate As we say, we will generate As we say, we will generate four tokens at a time, but in Deep four tokens at a time, but in Deep four tokens at a time, but in Deep Seeking they do it dynamically; I Seeking they do it dynamically; I Seeking they do it dynamically; I mean, you know, they decide on the fly, mean, you know, they decide on the fly, mean, you know, they decide on the fly, "Oh, I'll only generate two tokens, "Oh, I'll only generate two tokens, "Oh, I'll only generate two tokens, four tokens, or five tokens," four tokens, or five tokens," four tokens, or five tokens," depending on the acceptance criteria, depending on the acceptance criteria, depending on the acceptance criteria, GPU load, and all sorts of GPU load, and all sorts of GPU load, and all sorts of metrics. So, each model, each metrics. So, each model, each metrics. So, each model, each laboratory has its own way laboratory has its own way laboratory has its own way of implementing it. and you probably of implementing it. and you probably of implementing it. and you probably want to stick with that, but you can want to stick with that, but you can want to stick with that, but you can also experiment. For example, also experiment. For example, also experiment. For example, coin models…Coin Lab does not provide coin models…Coin Lab does not provide coin models…Coin Lab does not provide D Flash models for drafting, but they are D Flash models for drafting, but they are D Flash models for drafting, but they are provided by a third-party provided by a third-party provided by a third-party lab, but they work lab, but they work lab, but they work quite well with them, so you have to quite well with them, so you have to quite well with them, so you have to test it and see. test it and see. Thank you. Yes. I guess it freezes, right?

Yes. I guess it freezes, right? Yes, maybe I'll check under. Yes, maybe I'll check under. Yes. I could turn off the cluster. Oh my, I ca Yes. I could turn off the cluster. Oh my, I ca n't write. Oh my. n't write. Oh my. So it works fine. So it works fine. Refresh. Refresh. Let me recreate the card. Okay. Hmm, something is wrong. He copied the URL and opened a new He copied the URL and opened a new tab, see what happens. tab, see what happens. tab, see what happens. Let me open a new tab. I would Let me open a new tab. I would n't close this one. Yes. You know, in all the tests it didn't You know, in all the tests it didn't happen. It's today. Can I happen. It's today. Can I happen. It's today. Can I at least go through and go at least go through and go at least go through and go through things? Okay, through things? Okay, through things? Okay, at least I can scroll, at least I can scroll, at least I can scroll, scroll. You know, since the cluster scroll. You know, since the cluster scroll. You know, since the cluster has decided to take a break, I'll go has decided to take a break, I'll go has decided to take a break, I'll go through the content I was supposed to cover through the content I was supposed to cover . The main purpose of running this . The main purpose of running this . The main purpose of running this benchmark was to show where… This benchmark was to show where… This benchmark was to show where… This knee is, i.e., the place where knee is, i.e., the place where knee is, i.e., the place where you can see the maximum throughput.

you can see the maximum throughput. If we go back to this diagram where If we go back to this diagram where we see the maximum throughput, we see the maximum throughput, we see the maximum throughput, while also making sure that while also making sure that while also making sure that we see the lowest tokens and the lowest we see the lowest tokens and the lowest we see the lowest tokens and the lowest latency in generating tokens. And that latency in generating tokens. And that latency in generating tokens. And that 's essentially the limit that 's essentially the limit that 's essentially the limit that was 32 concurrency. was 32 concurrency. was 32 concurrency. Since above 32 Since above 32 Since above 32 concurrencies, concurrencies, concurrencies, token generation will be more or less the token generation will be more or less the token generation will be more or less the same. It won't change much. same. It won't change much. same. It won't change much. The throughput will be the same, but the The throughput will be the same, but the The throughput will be the same, but the latency will increase drastically, and we latency will increase drastically, and we latency will increase drastically, and we don't want that. This is more or less the don't want that. This is more or less the don't want that. This is more or less the limit you have reached. And this limit you have reached. And this limit you have reached. And this is important information that is important information that is important information that we need to remember and pay we need to remember and pay we need to remember and pay attention to when implementing attention to when implementing attention to when implementing these things. And if these things. And if so, you can go to 100. Look so, you can go to 100. Look at the spreadsheet. at the spreadsheet. 100? 100? Yes, go to 100. And then continue Yes, go to 100. And then continue Yes, go to 100. And then continue from from from Yes. Yes. Yes. 78. 78. Where is it? Here you go.

Where is it? Here you go. Okay, Okay, Okay, so…One works. That's good. Okay so…One works. That's good. Okay so…One works. That's good. Okay , great. So let's move on to the , great. So let's move on to the , great. So let's move on to the eighth. Let me eighth. Let me open up too. Who does this? All right. In module eight, we'll quickly go over what flags and knobs you can what flags and knobs you can what flags and knobs you can tweak to make sure you tweak to make sure you tweak to make sure you can exceed that limit can exceed that limit can exceed that limit we talked about at 32 we talked about at 32 we talked about at 32 concurrency. Do you see this concurrency. Do you see this concurrency. Do you see this limit and how to exceed it? And what to limit and how to exceed it? And what to limit and how to exceed it? And what to remember when thinking about it. remember when thinking about it. So, in this module I'm going to give you a So, in this module I'm going to give you a thinking model. It's all about thinking model. It's all about thinking model. It's all about measurement, which is why, with a lot of measurement, which is why, with a lot of measurement, which is why, with a lot of different cells, I'm constantly running different cells, I'm constantly running different cells, I'm constantly running stress tests. And it's very important stress tests. And it's very important stress tests. And it's very important to run these to run these to run these load tests to see what load tests to see what load tests to see what the performance is like and whether any the performance is like and whether any the performance is like and whether any changes that you make are impacting the changes that you make are impacting the changes that you make are impacting the performance? Is it getting better? Is performance? Is it getting better? Is performance? Is it getting better? Is it getting worse? And by what specific it getting worse? And by what specific it getting worse? And by what specific measure is it getting worse, right? For measure is it getting worse, right? For measure is it getting worse, right? For example, if example, if example, if the throughput is constantly increasing or the throughput is constantly increasing or the throughput is constantly increasing or staying the same but the staying the same but the staying the same but the latency is changing, then something is latency is changing, then something is latency is changing, then something is either correct or not. We must either correct or not. We must either correct or not. We must remember this. There are tons of different remember this. There are tons of different remember this. There are tons of different knobs you can adjust in knobs you can adjust in knobs you can adjust in vLLM to improve performance, but vLLM to improve performance, but vLLM to improve performance, but the four main ones are the four main ones are the four main ones are GPU memory utilization. Please ensure

GPU memory utilization. Please ensure GPU memory utilization. Please ensure it is set correctly it is set correctly it is set correctly depending on your use case. depending on your use case. depending on your use case. Again, if you're running multiple Again, if you're running multiple Again, if you're running multiple models on the same GPU, you might models on the same GPU, you might models on the same GPU, you might want to keep them low want to keep them low , and this is a trade-off you make , and this is a trade-off you make , and this is a trade-off you make . But if you . But if you . But if you only run one model and want to make only run one model and want to make only run one model and want to make sure it gets the best sure it gets the best sure it gets the best possible performance, you should possible performance, you should possible performance, you should increase this value to 0.9 or increase this value to 0.9 or increase this value to 0.9 or even 0.95. Another thing is to even 0.95. Another thing is to even 0.95. Another thing is to make sure that the maximum number of make sure that the maximum number of make sure that the maximum number of sequences is also set to an sequences is also set to an sequences is also set to an appropriate level. This must be appropriate level. This must be appropriate level. This must be determined through testing. In the specific determined through testing. In the specific determined through testing. In the specific implementation, if you look at implementation, if you look at implementation, if you look at vLLM.yaml, we start with 32 sequences, vLLM.yaml, we start with 32 sequences, vLLM.yaml, we start with 32 sequences, right? So this tells vLLM that it can right? So this tells vLLM that it can right? So this tells vLLM that it can only process 32 requests only process 32 requests only process 32 requests at a time and no more. And if at a time and no more. And if at a time and no more. And if you process more, you will see a drop in you process more, you will see a drop in you process more, you will see a drop in performance. There are many different flags, performance. There are many different flags, performance. There are many different flags, but these are key and worth but these are key and worth but these are key and worth remembering. Make sure remembering. Make sure remembering. Make sure you also set the model length you also set the model length you also set the model length to the correct one. And then, you know, to the correct one. And then, you know, to the correct one. And then, you know, make sure your batch tokens make sure your batch tokens make sure your batch tokens are the correct length as well are the correct length as well . You don't want to... You want to keep it . You don't want to... You want to keep it . You don't want to... You want to keep it larger if you want to larger if you want to larger if you want to do prefilling faster do prefilling faster so your GPU can handle it.

so your GPU can handle it. so your GPU can handle it. You want to make it smaller if You want to make it smaller if You want to make it smaller if your GPU is a bit slower and you can't your GPU is a bit slower and you can't your GPU is a bit slower and you can't handle it. But yes, the idea is to handle it. But yes, the idea is to handle it. But yes, the idea is to change one change one change one flag at a time. You don't want to change flag at a time. You don't want to change flag at a time. You don't want to change many things. And when you change something, many things. And when you change something, many things. And when you change something, run… Run benchmarks run… Run benchmarks run… Run benchmarks and see what and see what and see what performance you're getting, measure it, and then performance you're getting, measure it, and then performance you're getting, measure it, and then make assumptions about what you should do next. make assumptions about what you should do next. Hmm, let's go over this quickly. We have Hmm, let's go over this quickly. We have about 10 minutes left? Oh no. about 10 minutes left? Oh no. I think we lost the P PVC. I guess that's what I think we lost the P PVC. I guess that's what happened. happened. happened. Oh, he got another one. Yes, I need to Oh, he got another one. Yes, I need to Oh, he got another one. Yes, I need to change that, I think I'll repeat it. Let me just change that, I think I'll repeat it. Let me just change that, I think I'll repeat it. Let me just change it here. It was supposed to change it here. It was supposed to change it here. It was supposed to be sequential, but another be sequential, but another be sequential, but another Jupiter notebook crashed. So. Change this and it Jupiter notebook crashed. So. Change this and it Jupiter notebook crashed. So. Change this and it should work. Hmm, again, like should work. Hmm, again, like should work. Hmm, again, like I said, there are different knobs I said, there are different knobs I said, there are different knobs you can adjust and see you can adjust and see you can adjust and see what happens, but we'll get some kind of what happens, but we'll get some kind of what happens, but we'll get some kind of measurement. We'll measure the baseline, measurement. We'll measure the baseline, measurement. We'll measure the baseline, then change a few flags and then change a few flags and then change a few flags and see what happens next. Hmm, does it see what happens next. Hmm, does it see what happens next. Hmm, does it work? Okay, great. Here you go work? Okay, great. Here you go work? Okay, great. Here you go . Works. While this . Works. While this . Works. While this low-level test is running, does low-level test is running, does low-level test is running, does anyone have any other questions?

anyone have any other questions? anyone have any other questions? Sorry. Sorry. I noticed there is a noticeable I noticed there is a noticeable drop in performance when drop in performance when drop in performance when optimizing the model, like here, from 32 optimizing the model, like here, from 32 optimizing the model, like here, from 32 to 64, a huge drop. Is that when to 64, a huge drop. Is that when to 64, a huge drop. Is that when you decide we should scale the you decide we should scale the you decide we should scale the hardware, or is it something hardware, or is it something hardware, or is it something you're confident you can still you're confident you can still you're confident you can still fine-tune? fine-tune? Hmm. Is Hmm. Is this actually the ceiling? this actually the ceiling? this actually the ceiling? Yes. The ceiling is calculated using Yes. The ceiling is calculated using Yes. The ceiling is calculated using mathematical calculations. I think mathematical calculations. I think mathematical calculations. I think in the previous modules, this one, in the previous modules, this one, in the previous modules, this one, we discussed what the we discussed what the we discussed what the maximum tokens per maximum tokens per maximum tokens per second cap might look like, what the throughput might look like, second cap might look like, what the throughput might look like, second cap might look like, what the throughput might look like, and, you know, what the expectations should be and, you know, what the expectations should be and, you know, what the expectations should be for the implementation. And if for the implementation. And if for the implementation. And if you see these maximum expectations, then that's you see these maximum expectations, then that's you see these maximum expectations, then that's probably the maximum limit of probably the maximum limit of probably the maximum limit of your hardware. And then you your hardware. And then you your hardware. And then you probably need to get additional probably need to get additional probably need to get additional equipment or better equipment to equipment or better equipment to equipment or better equipment to do it. So yeah, there's a do it. So yeah, there's a do it. So yeah, there's a limit, you know, you can only tune limit, you know, you can only tune limit, you know, you can only tune as much as you can, but you're trying to as much as you can, but you're trying to as much as you can, but you're trying to get the most out of the GPU.

get the most out of the GPU. You want to share your slides before You want to share your slides before proceeding. Share slides proceeding. Share slides proceeding. Share slides very quickly. Server? I can very quickly. Server? I can very quickly. Server? I can leave it for a while, for a few hours. I'm leave it for a while, for a few hours. I'm leave it for a while, for a few hours. I'm not going to leave this just because not going to leave this just because not going to leave this just because you have access to a GPU. But what will I give you? you have access to a GPU. But what will I give you? you have access to a GPU. But what will I give you? If you scan this QR code, If you scan this QR code, If you scan this QR code, you'll get access to the you'll get access to the you'll get access to the PowerPoint presentation and also to the repository PowerPoint presentation and also to the repository PowerPoint presentation and also to the repository if you want to run it on your if you want to run it on your if you want to run it on your Akamai cloud account or run it Akamai cloud account or run it Akamai cloud account or run it on your laptop? You can do it. on your laptop? You can do it. on your laptop? You can do it. Scanning this QR code will take Scanning this QR code will take Scanning this QR code will take you to a page that offers all the you to a page that offers all the you to a page that offers all the resources you need. If you don't have an resources you need. If you don't have an resources you need. If you don't have an Akamai Cloud account, you can Akamai Cloud account, you can Akamai Cloud account, you can sign up and receive $300 sign up and receive $300 sign up and receive $300 in credit for a free account. That's in credit for a free account. That's in credit for a free account. That's all you need. And if all you need. And if all you need. And if you have any questions, you have any questions, you have any questions, you can find our contact details here. Homer, I'm Dwan you can find our contact details here. Homer, I'm Dwan you can find our contact details here. Homer, I'm Dwan Lightfoot. And yes, speaking of which Lightfoot. And yes, speaking of which , if you scan the QR code, I'll let , if you scan the QR code, I'll let , if you scan the QR code, I'll let Homer finish this. I Homer finish this. I Homer finish this. I just want to make sure you all just want to make sure you all just want to make sure you all get this before people start get this before people start get this before people start leaving. Thank you.

leaving. Thank you. Okay, we'll leave this for a few more Okay, we'll leave this for a few more minutes. Once everyone understands it. minutes. Once everyone understands it. Yes, go ahead. Mhm. Personal assistant? Yes, go ahead. Mhm. Personal assistant? Um, so So you can use Ollamas Um, so So you can use Ollamas Um, so So you can use Ollamas It depends on your equipment. I think that's the It depends on your equipment. I think that's the first thing I would first thing I would first thing I would look at. No problem. Thank you. look at. No problem. Thank you. look at. No problem. Thank you. Oh, you have both? For the sake of argument, Oh, you have both? For the sake of argument, Oh, you have both? For the sake of argument, let's assume you have the best let's assume you have the best let's assume you have the best GPU. Let's assume you have um Yes. GPU. Let's assume you have um Yes. GPU. Let's assume you have um Yes. Yes, you have the latest Yes, you have the latest Yes, you have the latest Blackwell GPU. Um, you know, I think for Blackwell GPU. Um, you know, I think for Blackwell GPU. Um, you know, I think for testing and doing, like testing and doing, like testing and doing, like figuring out what model you want to use for figuring out what model you want to use for figuring out what model you want to use for your use case, I think it your use case, I think it your use case, I think it 's good to use Ollama and Llama.cpp 's good to use Ollama and Llama.cpp 's good to use Ollama and Llama.cpp to quickly get it up and running and try it out to quickly get it up and running and try it out to quickly get it up and running and try it out and see what happens. Uh, but and see what happens. Uh, but and see what happens. Uh, but when you have a model that you know I'm when you have a model that you know I'm when you have a model that you know I'm definitely going to use in this definitely going to use in this definitely going to use in this use case and it works great, you're use case and it works great, you're use case and it works great, you're probably going to want to probably going to want to probably going to want to switch to uh vLLM, SGLang switch to uh vLLM, SGLang switch to uh vLLM, SGLang or one of these other or one of these other or one of these other runtime models. Uh, the reason runtime models. Uh, the reason runtime models. Uh, the reason is that they allow you to is that they allow you to is that they allow you to run at higher run at higher run at higher concurrency and give you concurrency and give you concurrency and give you more performance. So if more performance. So if more performance. So if you have a single agent running, you have a single agent running, you have a single agent running, I think you can use Ollam. This I think you can use Ollam. This I think you can use Ollam. This isn't a big deal since you isn't a big deal since you isn't a big deal since you only have one request at a time. But only have one request at a time. But only have one request at a time. But if let's say you're running 20 if let's say you're running 20 if let's say you're running 20 agents and you want to run them in agents and you want to run them in agents and you want to run them in parallel...Yes. Yes, you can

parallel...Yes. Yes, you can parallel...Yes. Yes, you can configure speculative decoding configure speculative decoding configure speculative decoding with vLLM. I don't think Ollama with vLLM. I don't think Ollama with vLLM. I don't think Ollama supports speculative decoding supports speculative decoding supports speculative decoding as far as I know, but I could be wrong. as far as I know, but I could be wrong. as far as I know, but I could be wrong. But yeah, if you want to get But yeah, if you want to get But yeah, if you want to get maximum performance out of your maximum performance out of your maximum performance out of your hardware, you definitely want to hardware, you definitely want to hardware, you definitely want to do this. Yes. I don't actually know if there do this. Yes. I don't actually know if there do this. Yes. I don't actually know if there is an LLM diffusion model. Is there an is an LLM diffusion model. Is there an open-source diffusion model currently available? Yes. Lana? Yes, I think open-source diffusion model currently available? Yes. Lana? Yes, I think vLLM supports it, so I think vLLM supports it, so I think vLLM supports it, so I think you'll be able to run it on you'll be able to run it on you'll be able to run it on vLLM. Ollama, if Ollama vLLM. Ollama, if Ollama vLLM. Ollama, if Ollama can handle it, then you can too. Well, yes. can handle it, then you can too. Well, yes. It's definitely possible. Okay. We 'll go back to the module and go through 'll go back to the module and go through it. Well, as I said, at 32 it. Well, as I said, at 32 it. Well, as I said, at 32 you see, you know, good performance, but you see, you know, good performance, but you see, you know, good performance, but at 64, well, the throughput at 64, well, the throughput at 64, well, the throughput stays the same, stays the same, stays the same, if not increases, but we see a significant if not increases, but we see a significant if not increases, but we see a significant increase in latency. And to get around that increase in latency. And to get around that increase in latency. And to get around that , one of the things I , one of the things I , one of the things I want to talk about is max want to talk about is max want to talk about is max sequence. This is approximately the number of sequence. This is approximately the number of sequence. This is approximately the number of requests that vLLM can requests that vLLM can requests that vLLM can process simultaneously. This sets process simultaneously. This sets process simultaneously. This sets that flag. If you want multiple that flag. If you want multiple that flag. If you want multiple requests to be processed requests to be processed requests to be processed in parallel, you need to increase this in parallel, you need to increase this in parallel, you need to increase this number. So, I think number. So, I think number. So, I think it's currently configured at 32, so it's currently configured at 32, so it's currently configured at 32, so we can increase that to 128. And then

we can increase that to 128. And then we can increase that to 128. And then we can run it, deploy it, and we can run it, deploy it, and we can run it, deploy it, and see how it impacts see how it impacts see how it impacts performance. OK, that changed a lot of performance. OK, that changed a lot of performance. OK, that changed a lot of things. Yes. Once that happens, things. Yes. Once that happens, things. Yes. Once that happens, we can run another test and we can run another test and you'll see that the limitation you'll see that the limitation we see here at 32 will disappear and the we see here at 32 will disappear and the we see here at 32 will disappear and the latency will be latency will be latency will be manageable again at higher manageable again at higher manageable again at higher concurrency levels as well. This is kind of a concurrency levels as well. This is kind of a concurrency levels as well. This is kind of a loop you want to get into loop you want to get into loop you want to get into . You want to find the right . You want to find the right . You want to find the right flags for each flags for each flags for each bottleneck location. Find the bottleneck location. Find the bottleneck location. Find the bottleneck, figure out why it's bottleneck, figure out why it's bottleneck, figure out why it's happening, what flag you can happening, what flag you can happening, what flag you can adjust to make sure adjust to make sure adjust to make sure you're sort of overcoming that you're sort of overcoming that you're sort of overcoming that bottleneck. Well, let's get started. This will take bottleneck. Well, let's get started. This will take bottleneck. Well, let's get started. This will take some time. Any questions? One more thing some time. Any questions? One more thing some time. Any questions? One more thing there. there.

Potentially impulsive movement Potentially impulsive movement Mhm. Mhm. Mhm. er, sometimes. Er, yes. Is er, sometimes. Er, yes. Is er, sometimes. Er, yes. Is there such a thing as a pulse-pounding there such a thing as a pulse-pounding there such a thing as a pulse-pounding transition to serverless? Is that transition to serverless? Is that transition to serverless? Is that the domain of, you know, the model routing layer the domain of, you know, the model routing layer the domain of, you know, the model routing layer on top? What is the pattern for on top? What is the pattern for on top? What is the pattern for Yes, erm, I mean Yes, erm, I mean Yes, erm, I mean auto-scaling is one of the auto-scaling is one of the auto-scaling is one of the patterns. More specifically, I will tell you that in patterns. More specifically, I will tell you that in patterns. More specifically, I will tell you that in LKE you can have dynamic, LKE you can have dynamic, LKE you can have dynamic, automatic scaling. They can automatic scaling. They can automatic scaling. They can scale up depending on your needs, scale up depending on your needs, scale up depending on your needs, depending on the dynamic traffic, depending on the dynamic traffic, depending on the dynamic traffic, you can scale up to, you know, maybe you can scale up to, you know, maybe you can scale up to, you know, maybe use a node with four use a node with four use a node with four GPUs instead of one because that can GPUs instead of one because that can GPUs instead of one because that can handle it better, right? handle it better, right? handle it better, right? So you can automate a lot of So you can automate a lot of So you can automate a lot of these things to make sure these things to make sure these things to make sure everything works well. The second everything works well. The second everything works well. The second part is that you can part is that you can part is that you can also build statistical models also build statistical models also build statistical models to make sure that you understand the to make sure that you understand the to make sure that you understand the traffic patterns over the long term, traffic patterns over the long term, traffic patterns over the long term, and the cluster can also dynamically and the cluster can also dynamically and the cluster can also dynamically predict the dynamic load in predict the dynamic load in predict the dynamic load in a sense and a sense and a sense and scale based on that. scale based on that. scale based on that. If you have a browser, I can If you have a browser, I can If you have a browser, I can show you where it is. show you where it is. show you where it is. Browser? Browser? Browser? Yes. If you open it, I will take you Yes. If you open it, I will take you Yes. If you open it, I will take you to it. to it. to it. All right. So this version has been All right. So this version has been All right. So this version has been successfully implemented.

successfully implemented. successfully implemented. Wow. Wow. Wow. And then we'll do a quick And then we'll do a quick And then we'll do a quick load test and load test and load test and we'll see more once we run it. All right. Do you have we'll see more once we run it. All right. Do you have we'll see more once we run it. All right. Do you have any other questions? Most people any other questions? Most people any other questions? Most people have left, but have left, but have left, but yes. Light foot. H. Yes. yes. Light foot. H. Yes. yes. Light foot. H. Yes. 32, that's where we saw the limit last time. 32, that's where we saw the limit last time. 32, that's where we saw the limit last time. Yes. Yes. Yes. Once we get to 64 we should Once we get to 64 we should Once we get to 64 we should see that things should improve. see that things should improve. see that things should improve. Okay. So there's been a big, Okay. So there's been a big, Okay. So there's been a big, drastic improvement. Throughput drastic improvement. Throughput drastic improvement. Throughput has actually increased quite significantly. Um has actually increased quite significantly. Um , so that's 1100 tokens per , so that's 1100 tokens per , so that's 1100 tokens per second. Whereas before, second. Whereas before, second. Whereas before, if we look at it here, at 64 if we look at it here, at 64 if we look at it here, at 64 we had 800 tokens per second and the we had 800 tokens per second and the we had 800 tokens per second and the latency was uh 2000 ms. But latency was uh 2000 ms. But latency was uh 2000 ms. But now, with the change now, with the change now, with the change we made, uh it's uh 1100 we made, uh it's uh 1100 we made, uh it's uh 1100 tokens per second and the latency tokens per second and the latency tokens per second and the latency is uh 800 ms. So it fell.

is uh 800 ms. So it fell. is uh 800 ms. So it fell. The latency dropped quite dramatically The latency dropped quite dramatically , and then we also increased , and then we also increased the throughput quite dramatically. So these are the the throughput quite dramatically. So these are the kinds of fine-tunings that can be kinds of fine-tunings that can be kinds of fine-tunings that can be done over time to make sure done over time to make sure done over time to make sure everything is working well once you have everything is working well once you have everything is working well once you have everything up and running. This can be everything up and running. This can be everything up and running. This can be done for a specific use case. done for a specific use case. done for a specific use case. You might not have six You might not have six , you know, 64 concurrent requests , you know, 64 concurrent requests , you know, 64 concurrent requests coming in in your particular coming in in your particular coming in in your particular use case, but maybe you do. So you need to use case, but maybe you do. So you need to use case, but maybe you do. So you need to make a decision based on make a decision based on make a decision based on your use cases. your use cases. your use cases. If concurrency is one If concurrency is one If concurrency is one higher than four, was that a higher than four, was that a higher than four, was that a trade-off you just made? trade-off you just made? trade-off you just made? Concurrency one? I mean, it Concurrency one? I mean, it Concurrency one? I mean, it could be a little higher or lower. It could be a little higher or lower. It could be a little higher or lower. It won't be the same all the time, won't be the same all the time, won't be the same all the time, but for example, before it was 37 and but for example, before it was 37 and but for example, before it was 37 and now it's 35. So it's a slight now it's 35. So it's a slight now it's 35. So it's a slight deviation that happens when the deviation that happens when the deviation that happens when the models are run. But there should models are run. But there should models are run. But there should be plus or minus two two tokens. We be plus or minus two two tokens. We be plus or minus two two tokens. We are now also running the full are now also running the full are now also running the full precision model. We no longer run FPA precision model. We no longer run FPA precision model. We no longer run FPA because we had to change notebooks because we had to change notebooks . That's another reason why . That's another reason why . That's another reason why you see it go from about 50 you see it go from about 50 you see it go from about 50 tokens to about 30 tokens. Uh, tokens to about 30 tokens. Uh, tokens to about 30 tokens. Uh, so that's the performance degradation we're so that's the performance degradation we're so that's the performance degradation we're seeing there. But but yes, that's all I seeing there. But but yes, that's all I seeing there. But but yes, that's all I have. This is one flag you have. This is one flag you have. This is one flag you can customize. There are also can customize. There are also can customize. There are also plenty of other flags that I implore plenty of other flags that I implore plenty of other flags that I implore everyone to look out for.

everyone to look out for. everyone to look out for. The workshop is going through this too. The workshop is going through this too. But, I think we're on time and I'll But, I think we're on time and I'll give it to Dwan. give it to Dwan. Yes. Thank you all for participating in Yes. Thank you all for participating in today's session. If you haven't today's session. If you haven't today's session. If you haven't scanned the QR code, the code, scanned the QR code, the code, scanned the QR code, the code, repository, and PowerPoint presentations repository, and PowerPoint presentations repository, and PowerPoint presentations are available on the website. I invite you to are available on the website. I invite you to are available on the website. I invite you to create a free create a free create a free Akamai cloud account. My name is Dwan Akamai cloud account. My name is Dwan Akamai cloud account. My name is Dwan Lightfoot and I'm joined by my Lightfoot and I'm joined by my Lightfoot and I'm joined by my colleague Omer and thank you all colleague Omer and thank you all colleague Omer and thank you all for joining us today. for joining us today. for joining us today. Feel free to ask us questions Feel free to ask us questions Feel free to ask us questions after the session. Thank you.