Is a Faster Model Actually Faster? Test the Whole Job — Morgan Suo, MiniMax

I'm Morgan from Minimax. Um, I'm Morgan from Minimax. Um, how many of you knew how many of you knew how many of you knew about Minimax before? Oh. Wow about Minimax before? Oh. Wow . I like it. . I like it. . I like it. Yes, because our Yes, because our Yes, because our company has been in existence for company has been in existence for company has been in existence for almost three years. But almost three years. But almost three years. But when I first entered the when I first entered the when I first entered the American American American market, almost no market, almost no market, almost no one knew us. Although at one knew us. Although at one knew us. Although at that time we had the that time we had the that time we had the first model for first model for first model for video generation. So at video generation. So at video generation. So at the end of the talk, I'll the end of the talk, I'll the end of the talk, I'll show you something. But show you something. But show you something. But first, I'll first, I'll first, I'll show off our show off our show off our newest large language newest large language newest large language model, which will be released model, which will be released model, which will be released next week. next week. Today my topic is the hidden costs of a the hidden costs of a the hidden costs of a faster model. When faster model. When faster model. When people think about an AI people think about an AI application application application embedded in their embedded in their embedded in their own own own workflow. When a workflow. When a workflow. When a new model comes out new model comes out new model comes out and I say, “Oh, and I say, “Oh, and I say, “Oh, please, we have a new please, we have a new please, we have a new model. "Would you like model. "Would you like model. "Would you like to try?" It to try?" It to try?" It generates tasks generates tasks generates tasks faster, but I think you're faster, but I think you're faster, but I think you're asking, "Is it worth asking, "Is it worth asking, "Is it worth switching to it?" switching to it?" You launch a You launch a research research research task and the report task and the report task and the report arrives quickly. But it arrives quickly. But it arrives quickly. But it misses a misses a misses a very important source.

very important source. very important source. So someone has to So someone has to So someone has to find this find this find this source and correct the source and correct the source and correct the report. I think this is what report. I think this is what report. I think this is what we do every day. Has we do every day. Has we do every day. Has work really become work really become work really become more efficient with this more efficient with this more efficient with this faster model? faster model? Today we Today we will look at how we will look at how we will look at how we reduce reduce reduce model computation. And then model computation. And then model computation. And then we ask how to assess we ask how to assess we ask how to assess the value of these changes in the the value of these changes in the the value of these changes in the model. Let me model. Let me model. Let me show you the problem show you the problem show you the problem we will use we will use we will use to answer this to answer this to answer this question. At Minimax, we question. At Minimax, we question. At Minimax, we create various create various create various multimodal multimodal multimodal models and products models and products models and products for users. for users. Quick chat responses Quick chat responses and research and research and research reports have different reports have different reports have different needs. So the task needs. So the task needs. So the task tells us what tells us what tells us what good good good performance means. performance means. performance means. Today I Today I Today I will focus on our will focus on our will focus on our newest language newest language newest language model M 3.1 Flash and model M 3.1 Flash and model M 3.1 Flash and one one one research research research task. So, here's task. So, here's task. So, here's the analyst's question. the analyst's question. As demand for AI servers grows, are servers grows, are board companies turning this board companies turning this growth into growth into growth into profits? PCB stands for profits? PCB stands for profits? PCB stands for printed circuit board.

printed circuit board. printed circuit board. I think you know what it is I think you know what it is I think you know what it is . It connects . It connects . It connects electronic electronic electronic components inside components inside components inside devices such as devices such as devices such as AI servers. Our AI servers. Our AI servers. Our research case research case research case looks at companies in looks at companies in looks at companies in this this this supply chain. The supply chain. The supply chain. The internal review internal review internal review describes the work on 14 describes the work on 14 describes the work on 14 company reports. company reports. company reports. The model must perform The model must perform The model must perform several tasks. She several tasks. She several tasks. She must find the must find the must find the necessary reports. It necessary reports. It necessary reports. It must compare the must compare the must compare the same reporting same reporting same reporting periods. And it periods. And it periods. And it must distinguish between must distinguish between must distinguish between different businesses. Then different businesses. Then different businesses. Then she has to explain what she has to explain what she has to explain what these numbers mean. these numbers mean. Think of Think of an analyst an analyst an analyst receiving a report. They receiving a report. They receiving a report. They need to discover need to discover need to discover the source of an important the source of an important the source of an important statement. And they statement. And they statement. And they need to check need to check need to check the calculations. They the calculations. They the calculations. They need to see what need to see what need to see what evidence might evidence might evidence might change the conclusion. change the conclusion. change the conclusion. So, a useful report So, a useful report So, a useful report provides the analyst with data provides the analyst with data provides the analyst with data that he can that he can that he can verify. And this is our verify. And this is our verify. And this is our standard for standard for standard for completed completed completed research work research work . So, one result . So, one result . So, one result shows why these shows why these shows why these checks are important. checks are important. This internal review This internal review highlighted a very highlighted a very highlighted a very useful distinction useful distinction . Yes, profits may be . Yes, profits may be . Yes, profits may be growing, but growing, but growing, but cash flow cash flow cash flow suggests otherwise.

suggests otherwise. The company sells The company sells more, so its more, so its more, so its reported profit reported profit reported profit increases, but customers increases, but customers increases, but customers haven't paid yet. Or haven't paid yet. Or haven't paid yet. Or she purchased she purchased she purchased materials that are still materials that are still materials that are still stored in the stored in the stored in the warehouse. So analytics warehouse. So analytics warehouse. So analytics needs to know why. needs to know why. needs to know why. The model should indicate the The model should indicate the cash flow statement and cash flow statement and related related related disclosures. So, disclosures. So, disclosures. So, a report is a report is a report is trustworthy when its trustworthy when its trustworthy when its claims can be claims can be claims can be verified. And the verified. And the verified. And the internal review here internal review here internal review here describes the research describes the research describes the research result. Therefore, we result. Therefore, we result. Therefore, we still need to still need to still need to check the sources check the sources check the sources and calculations, as well as and calculations, as well as and calculations, as well as measure the time and measure the time and measure the time and cost of this same cost of this same cost of this same work. But the technical work. But the technical work. But the technical challenges are complex. challenges are complex. challenges are complex. You divide You divide You divide the tasks. They the tasks. They the tasks. They help us help us help us understand specific understand specific understand specific trade-offs. With that trade-offs. With that trade-offs. With that task clear, task clear, task clear, let's let's let's see what's see what's see what's changed in Gemini 1.5 Flash. changed in Gemini 1.5 Flash. This table shows the This table shows the main changes in Gemini 1.5 main changes in Gemini 1.5 main changes in Gemini 1.5 Flash. I will link them to Flash. I will link them to Flash. I will link them to a few simple a few simple a few simple questions. First questions. First , where can we , where can we , where can we reduce reduce reduce the load on the the load on the the load on the attention mechanism?

attention mechanism? Second, where can we Second, where can we Second, where can we store and store and store and transfer large transfer large transfer large amounts of data? amounts of data? Third, how does Third, how does service setup service setup affect affect user wait time? And fourth, we fourth, we fourth, we can also show can also show can also show how intensively how intensively how intensively the model is reasoning. These the model is reasoning. These the model is reasoning. These choices affect choices affect choices affect different parts of the job, different parts of the job, different parts of the job, and each needs to be and each needs to be and each needs to be checked for checked for checked for compliance with the compliance with the compliance with the desired outcome. desired outcome. desired outcome. So every So every So every optimization should optimization should optimization should contribute to getting the contribute to getting the contribute to getting the job done. Let's start with job done. Let's start with job done. Let's start with how the model how the model how the model uses long uses long uses long input data. Today input data. Today , it seems like every , it seems like every , it seems like every model says, "Oh, model says, "Oh, model says, "Oh, I have a really big I have a really big I have a really big context window." As context window." As context window." As we said, we have we said, we have we said, we have 1 million tokens. So 1 million tokens. So 1 million tokens. So among this million, among this million, among this million, how can it how can it how can it help you find the help you find the help you find the result quickly, result quickly, result quickly, faster and correctly? faster and correctly? Some passages in this Some passages in this report carry report carry report carry much more weight than much more weight than much more weight than others for a particular others for a particular others for a particular issue. Sparse issue. Sparse issue. Sparse attention means that we attention means that we attention means that we use very use very use very small samples, like small samples, like that blue that blue that blue rectangle. It rectangle. It rectangle. It evaluates blocks of text, evaluates blocks of text, evaluates blocks of text, and the underlying and the underlying and the underlying attention mechanism then calculates attention mechanism then calculates attention mechanism then calculates the value of the selected the value of the selected the value of the selected blocks. The model also blocks. The model also blocks. The model also keeps the keeps the keeps the surrounding text in view. This surrounding text in view. This surrounding text in view. This sample actually sample actually sample actually varies depending varies depending varies depending on the query, and on the query, and on the query, and different questions may different questions may different questions may require different require different require different context snippets.

context snippets. The number at the bottom shows The number at the bottom shows how much text how much text how much text this this this published published published Flash example contains. And the total Flash example contains. And the total Flash example contains. And the total input volume could input volume could input volume could be much be much be much larger. In our larger. In our larger. In our research research research project, the goal is project, the goal is to reduce this to reduce this to reduce this workload by focusing on workload by focusing on workload by focusing on many of these many of these many of these documents. But documents. But documents. But even within a single even within a single even within a single document, a short document, a short document, a short footnote can change the footnote can change the footnote can change the entire conclusion. So, entire conclusion. So, entire conclusion. So, for a for a for a cash flow question, a cash flow question, a cash flow question, a footnote can carry footnote can carry footnote can carry more weight than pages of more weight than pages of more weight than pages of industry forecasts. industry forecasts. industry forecasts. Therefore, we want the Therefore, we want the Therefore, we want the model to take model to take model to take this passage into account as well. A smaller this passage into account as well. A smaller this passage into account as well. A smaller amount of attentional work amount of attentional work amount of attentional work should still should still should still find important find important find important evidence. So, the evidence. So, the evidence. So, the next question is next question is next question is simple. Can we simple. Can we simple. Can we still find the still find the still find the facts we need? M 3.1 facts we need? M 3.1 facts we need? M 3.1 flash also flash also flash also uses this uses this uses this sparse attention in sparse attention in sparse attention in its first three its first three its first three layers. This makes layers. This makes layers. This makes the search an important the search an important the search an important check. Therefore, this check. Therefore, this check. Therefore, this separate task separate task separate task places facts at places facts at places facts at different positions in the different positions in the different positions in the long input data.

long input data. So, in one run, So, in one run, the model runs all the model runs all the model runs all 39 queries, and the longest 39 queries, and the longest 39 queries, and the longest input data had input data had input data had about 830 thousand about 830 thousand about 830 thousand tokens. Therefore, the tokens. Therefore, the tokens. Therefore, the result covers result covers result covers all cases in this all cases in this all cases in this problem. And for problem. And for problem. And for our our our research research research project, we need project, we need project, we need tasks with real tasks with real tasks with real reports. Can reports. Can reports. Can the model find a the model find a the model find a footnote about cash footnote about cash footnote about cash flows? Is flows? Is flows? Is it using the it using the it using the correct reporting correct reporting correct reporting period? And can it period? And can it period? And can it detect contradictions detect contradictions detect contradictions between different between different between different documents? So, documents? So, documents? So, move this move this move this important passage to important passage to important passage to different places in the input different places in the input different places in the input data and data and data and ask the same ask the same ask the same question again. Long question again. Long question again. Long input text is input text is input text is useful when the model useful when the model useful when the model can find a key can find a key can find a key fact. So, looking for fact. So, looking for fact. So, looking for evidence is also evidence is also evidence is also one of the one of the one of the verification methods. Next, we verification methods. Next, we verification methods. Next, we look at the numbers look at the numbers look at the numbers used used used within this within this within this model. The model model. The model model. The model stores internal stores internal stores internal information from information from information from the previous text, the previous text, the previous text, which we call the which we call the which we call the KB cache. And M 3.1 flash KB cache. And M 3.1 flash KB cache. And M 3.1 flash changes its keys and changes its keys and changes its keys and values ​​from 8 bits to 4 values ​​from 8 bits to 4 values ​​from 8 bits to 4 bits. However, the request bits. However, the request bits. However, the request remains at the remains at the remains at the 8-bit level. And it can 8-bit level. And it can 8-bit level. And it can reduce the amount of data reduce the amount of data reduce the amount of data we store and we store and we store and move. This also move. This also move. This also changes the position of this changes the position of this changes the position of this internal internal internal information.

information. information. Take a look at that Take a look at that Take a look at that article. The average scores are article. The average scores are article. The average scores are actually almost actually almost actually almost the same. But the the same. But the the same. But the detailed score there detailed score there detailed score there drops by about 6.6% drops by about 6.6% . Although some other scores . Although some other scores . Although some other scores are increasing. So, the are increasing. So, the are increasing. So, the average score average score average score can actually can actually can actually hide a bug hide a bug hide a bug that is worrying your that is worrying your that is worrying your user. This user. This user. This compares two complete compares two complete compares two complete configurations, but doesn't configurations, but doesn't configurations, but doesn't tell us exactly what tell us exactly what tell us exactly what changes affected changes affected changes affected each estimate. Therefore, each estimate. Therefore, each estimate. Therefore, for our for our for our research research research project, we need to project, we need to project, we need to check the result again check the result again check the result again . Are . Are . Are the sources correct? Are the sources correct? Are the sources correct? Are the calculations correct? And does the calculations correct? And does the calculations correct? And does the explanation fit the the explanation fit the the explanation fit the available evidence? The same available evidence? The same available evidence? The same ideas ideas ideas apply to the apply to the apply to the rest of the model. rest of the model. rest of the model. These are experts. Our These are experts. Our These are experts. Our model has an model has an model has an MOE architecture. It MOE architecture. It MOE architecture. It has computational has computational has computational components that we components that we components that we call experts call experts , that is, MOE experts. , that is, MOE experts. So, she chooses So, she chooses some of them for some of them for some of them for each token. In M 3.1 each token. In M 3.1 each token. In M 3.1 flash, these selected flash, these selected flash, these selected experts experts experts use 4- use 4- bit weights and bit weights and bit weights and activations. The joint activations. The joint activations. The joint expert is not subject expert is not subject expert is not subject to this change. Weights are the to this change. Weights are the to this change. Weights are the stored values stored values stored values of the model, and activations of the model, and activations are the values ​​it are the values ​​it are the values ​​it calculates. Lower calculates. Lower calculates. Lower precision allows for precision allows for precision allows for less less less data movement and more data movement and more data movement and more efficient efficient efficient calculations. However, the calculations. However, the calculations. However, the full calculation full calculation full calculation consists of consists of consists of several stages. We several stages. We several stages. We scale the numbers, scale the numbers, scale the numbers, perform calculations, perform calculations, perform calculations, and restore and restore and restore the scale.

the scale. the scale. The sequence of these The sequence of these The sequence of these steps is steps is steps is critical. The table critical. The table critical. The table shows exactly where the shows exactly where the rescaling occurs . The correct order of . The correct order of . The correct order of these actions is the key to these actions is the key to these actions is the key to getting the correct getting the correct getting the correct answer. So, FastMat answer. So, FastMat answer. So, FastMat should remain should remain should remain correct. When we correct. When we correct. When we check the full check the full check the full result, we result, we result, we evaluate the implementation evaluate the implementation evaluate the implementation as a whole. For this as a whole. For this as a whole. For this research research research task, we task, we task, we ask the same ask the same ask the same questions again. We questions again. We questions again. We check if check if check if the answers still the answers still the answers still fail our tests. fail our tests. Now let's take a look Now let's take a look inside. We inside. We inside. We have already considered the internal have already considered the internal have already considered the internal structure of the model. structure of the model. structure of the model. Now let's see how Now let's see how Now let's see how it generates it generates it generates the answer for you. A the answer for you. A maintenance method maintenance method called called called speculative speculative speculative decoding works in decoding works in decoding works in two stages. A faster two stages. A faster two stages. A faster draft process draft process draft process helps helps helps to propose a to propose a to propose a short short short sequence of sequence of sequence of tokens. And then the tokens. And then the tokens. And then the target model target model target model checks them. The system checks them. The system checks them. The system stores the part stores the part stores the part that the model accepts and that the model accepts and that the model accepts and makes adjustments makes adjustments makes adjustments when necessary.

when necessary. After which the whole After which the whole process is repeated. process is repeated. process is repeated. Sometimes it keeps Sometimes it keeps Sometimes it keeps a few of the a few of the a few of the proposed proposed proposed tokens, and sometimes tokens, and sometimes tokens, and sometimes the draft works in a way the draft works in a way the draft works in a way that makes it impossible to that makes it impossible to that makes it impossible to leave them. So, leave them. So, leave them. So, the possibility is to the possibility is to the possibility is to guess guess guess multiple tokens in a multiple tokens in a multiple tokens in a single check by the single check by the single check by the target model. target model. target model. However, drafting However, drafting However, drafting , reviewing, , reviewing, , reviewing, and rejecting these and rejecting these and rejecting these tokens also tokens also tokens also takes time. So takes time. So takes time. So also consider the time also consider the time also consider the time spent spent spent proofreading the draft. proofreading the draft. proofreading the draft. This verifies the This verifies the This verifies the proposed tokens proposed tokens . However, financial . However, financial . However, financial statements and statements and statements and sources require sources require sources require separate verification. separate verification. separate verification. Therefore, for our Therefore, for our Therefore, for our research work, research work, research work, we also need to we also need to we also need to include include include source searches, source searches, source searches, tool calls, and tool calls, and tool calls, and retries. The retries. The retries. The final indicator final indicator final indicator is the time to receive is the time to receive is the time to receive a report that the analyst a report that the analyst a report that the analyst can actually can actually can actually use in use in use in his work. Even his work. Even his work. Even if a single query if a single query if a single query performs well, a performs well, a performs well, a large number of large number of large number of users can users can users can change the experience. change the experience. Here is a separate Here is a separate maintenance task. It maintenance task. It maintenance task. It uses uses uses story generation with a story generation with a story generation with a low level of low level of low level of logical effort. But logical effort. But logical effort. But the number of simultaneous the number of simultaneous the number of simultaneous requests increases, requests increases, requests increases, for example, from 32 to 64.

for example, from 32 to 64. for example, from 32 to 64. The total output The total output The total output power per power per power per second increases second increases second increases by about 1.9 times. by about 1.9 times. by about 1.9 times. But look at the But look at the But look at the wait time. wait time. wait time. In fact, the In fact, the In fact, the P95 wait time for the P95 wait time for the P95 wait time for the first token first token first token is increasing. It is increasing. It is increasing. It increases increases increases from about 1.5 from about 1.5 from about 1.5 seconds to 11.3 seconds seconds to 11.3 seconds seconds to 11.3 seconds (instead of 0.6 seconds). (instead of 0.6 seconds). (instead of 0.6 seconds). So, for 95% of these So, for 95% of these So, for 95% of these requests, the first token requests, the first token requests, the first token arrives by this arrives by this arrives by this point. And now point. And now point. And now the wait is the wait is the wait is eight times longer. eight times longer. So, the average So, the average wait time is an wait time is an wait time is an indicator that we indicator that we indicator that we can be can be can be happy with, but we happy with, but we happy with, but we shouldn't look shouldn't look shouldn't look at it alone. Because if we at it alone. Because if we at it alone. Because if we focus only on focus only on focus only on it, it can it, it can it, it can radically change the radically change the radically change the entire perception. entire perception. entire perception. Users still Users still Users still experience latency experience latency experience latency even when the even when the even when the throughput throughput throughput is, say, is, say, is, say, over 300 requests per over 300 requests per over 300 requests per second. But you second. But you second. But you still need to care still need to care still need to care about the final about the final about the final wait time. The first wait time. The first wait time. The first token may be the token may be the token may be the result of result of result of reasoning, but we reasoning, but we reasoning, but we need to measure need to measure need to measure the moment of receiving the the moment of receiving the the moment of receiving the finished report. So finished report. So finished report. So set a deadline set a deadline set a deadline for your report, and then for your report, and then for your report, and then choose the parameters choose the parameters choose the parameters that will help you that will help you that will help you meet it. There is meet it. There is meet it. There is another setting another setting another setting that changes the amount of that changes the amount of that changes the amount of work we work we work we assign to the model. This is the assign to the model. This is the assign to the model. This is the first time first time first time we have used we have used we have used this kind of this kind of this kind of reasoning control. This part reasoning control. This part reasoning control. This part allows you to control allows you to control allows you to control the maximum the maximum the maximum changes in our changes in our changes in our query. This also query. This also query. This also controls how many

controls how many controls how many tokens we allocate tokens we allocate tokens we allocate to the model's "thinking" process to the model's "thinking" process to the model's "thinking" process . In the . In the document settings, "thinking" document settings, "thinking" is enabled, but if is enabled, but if is enabled, but if you leave the effort field you leave the effort field you leave the effort field empty, the empty, the empty, the default is default is default is "maximum." So, the "maximum." So, the "maximum." So, the default setting default setting default setting can can can change the amount of change the amount of work we request. Here is an example programming problem that programming problem that illustrates this illustrates this illustrates this trade-off. trade-off. trade-off. Let's compare the high Let's compare the high Let's compare the high level of reasoning and the level of reasoning and the level of reasoning and the maximum. maximum. maximum. The high level The high level The high level uses fewer uses fewer uses fewer source tokens than source tokens than source tokens than the maximum level, but the maximum level, but the maximum level, but the accuracy the accuracy the accuracy has dropped from 80% to 68%. Consequently, has dropped from 80% to 68%. Consequently, has dropped from 80% to 68%. Consequently, fewer fewer fewer tokens resulted in tokens resulted in tokens resulted in lower accuracy in lower accuracy in lower accuracy in completing these completing these completing these tasks. Give tasks. Give tasks. Give this task the this task the this task the “thinking” budget “thinking” budget “thinking” budget it really it really it really needs. For our needs. For our needs. For our research, research, research, compare these compare these compare these settings on settings on settings on identical reports and identical reports and identical reports and see if each see if each see if each one finds evidence one finds evidence one finds evidence and confirms and confirms and confirms the calculations. Other budget constraints may be required for a the calculations. Other budget constraints may be required for a the calculations. Other budget constraints may be required for a quick initial quick initial quick initial review of the report review of the report . We . We need to record need to record need to record this setting— this setting— which which which reasoning control we chose— reasoning control we chose— so that we can use so that we can use so that we can use it in future it in future it in future work. OK.

work. OK. work. OK. We then implement this We then implement this We then implement this research research research workflow with workflow with workflow with all new all new all new model updates. model updates. First, we can First, we can fix issues fix issues fix issues with the original reports, with the original reports, with the original reports, use use use the same the same the same cut-off date for cut-off date for cut-off date for information, and information, and information, and agree on agree on agree on report acceptance criteria report acceptance criteria report acceptance criteria before starting before starting before starting the comparison. Next, we the comparison. Next, we the comparison. Next, we can check can check can check three things. First: three things. First: three things. First: can the analyst can the analyst can the analyst trust this report? trust this report? trust this report? Discover the sources on Discover the sources on Discover the sources on which which which important statements are based, important statements are based, important statements are based, check reporting check reporting check reporting periods, list periods, list periods, list financial figures, financial figures, financial figures, and look for evidence and look for evidence and look for evidence that could change that could change that could change the conclusions. Second: what the conclusions. Second: what the conclusions. Second: what was the cost of was the cost of was the cost of completing the completed completing the completed completing the completed work? How long did work? How long did work? How long did it take? it take? it take? Consider Consider Consider data retrieval, data retrieval, data retrieval, tool costs, and tool costs, and tool costs, and number of number of number of retries. So, if retries. So, if retries. So, if a person has to a person has to a person has to proofread the report proofread the report proofread the report before use, before use, before use, consider this consider this consider this work as well. So, work as well. So, work as well. So, include all include all include all failed attempts in failed attempts in failed attempts in the cost of generating the cost of generating the cost of generating these acceptable these acceptable these acceptable reports. So reports. So reports. So test test test the number of expected the number of expected the number of expected new users and new users and new users and check if the reports are check if the reports are check if the reports are ready on time. And ready on time. And ready on time. And measure the value measure the value measure the value that reporting can that reporting can that reporting can actually actually actually deliver. And third: deliver. And third: deliver. And third: can we can we can we repeat this repeat this repeat this result? Save result? Save result? Save this model and runtime versions this model and runtime versions , and save the , and save the query, settings, query, settings, query, settings, and sources. So and sources. So and sources. So keep these keep these keep these error examples, we error examples, we error examples, we can start with a can start with a can start with a small small small deployment and deployment and deployment and leave the path back.

leave the path back. So, before we wrap So, before we wrap up, here's a up, here's a up, here's a quick recap of quick recap of quick recap of today's today's today's conversation. It was conversation. It was conversation. It was created created created using MiniMax A3, using MiniMax A3, using MiniMax A3, our video AI model our video AI model , and you can , and you can , and you can check it out. So, this is check it out. So, this is check it out. So, this is another model another model another model that we are working on, and that we are working on, and that we are working on, and it is with open it is with open it is with open scales. So, scales. So, scales. So, anyone who wants to anyone who wants to anyone who wants to create this really create this really create this really only has one problem with only has one problem with only has one problem with video over text. video over text. video over text. OK. So, OK. So, OK. So, let's go back to let's go back to let's go back to our opening our opening our opening question today. question today. question today. Should we switch to a Should we switch to a Should we switch to a faster model? And faster model? And faster model? And look at this look at this look at this finished work that finished work that finished work that we've been talking about for the we've been talking about for the we've been talking about for the last 18 minutes. last 18 minutes. last 18 minutes. So, for our So, for our So, for our analyst, this means a analyst, this means a analyst, this means a report with resources report with resources report with resources that can be opened and that can be opened and that can be opened and numbers that can be numbers that can be numbers that can be verified. It should verified. It should verified. It should come when it is come when it is come when it is needed. And for needed. And for needed. And for our M 3.1 flash, this our M 3.1 flash, this our M 3.1 flash, this changes the computational cost changes the computational cost changes the computational cost , and the full , and the full , and the full task shows what task shows what task shows what this means for this means for this means for the user. So, the user. So, the user. So, measure how much it measure how much it measure how much it costs us to complete the costs us to complete the costs us to complete the work on time. So work on time. So work on time. So faster models, I faster models, I faster models, I think faster models think faster models think faster models really really really matter. But the matter. But the matter. But the most important thing is what most important thing is what most important thing is what we can do with them we can do with them we can do with them . Thank you, . Thank you, . Thank you, friends.