Debugging Across Two Runtimes and Two Different Languages | Alex Drum (Hytale) | GameDev Day 2026

And now we'll invite And now we'll invite Alex Stram. Are you here, Alex Stram. Are you here, Alex Stram. Are you here, Alex? Alex? Alex? Hello, how are you? Hello, how are you? Hello, how are you? Okay, yes. Okay, yes. Okay, yes. How are you? I'm How are you? I'm How are you? I'm fine, you know. fine, you know. Perfectly. So, yes. Perfectly. So, yes. Alex, we can hear you well Alex, we can hear you well Alex, we can hear you well . We're giving . We're giving . We're giving you the floor, and I understand you the floor, and I understand that you're going to tell us that you're going to tell us that you're going to tell us about debugging in about debugging in about debugging in two two two runtimes and in two runtimes and in two runtimes and in two different languages. So, different languages. So, different languages. So, the word is yours. the word is yours. the word is yours. Perfectly. Thanks, guys. Perfectly. Thanks, guys. Perfectly. Thanks, guys. Hello everyone. I'll Hello everyone. I'll Hello everyone. I'll put my put my put my introduction aside for introduction aside for introduction aside for a moment and start with a a moment and start with a a moment and start with a little little little demonstration of what this is all about demonstration of what this is all about demonstration of what this is all about . . . Look at this Look at this Look at this character who just character who just character who just walked across the screen and walked across the screen and walked across the screen and hit the chest. And hit the chest. And hit the chest. And here there is a mistake. here there is a mistake. This mistake is hard This mistake is hard to notice because you'd to notice because you'd to notice because you'd think: I opened think: I opened think: I opened the chest and got two the chest and got two the chest and got two items. I guess it's items. I guess it's items. I guess it's scaling, right? I was scaling, right? I was scaling, right? I was probably planning on probably planning on probably planning on getting two getting two getting two items. The inventory items. The inventory items. The inventory says I says I says I received two items, received two items, received two items, but but but only one should have dropped from the chest.

only one should have dropped from the chest. So now I want So now I want you to make a decision. you to make a decision. Is this error on the client Is this error on the client or on the server? or on the server? Keep this thought in mind Keep this thought in mind for a moment, because we for a moment, because we for a moment, because we will return to it will return to it will return to it after a brief after a brief after a brief introduction. So, which introduction. So, which introduction. So, which side is broken? side is broken? Client or server? Client or server? This question is This question is This question is much much much more difficult to answer than it more difficult to answer than it more difficult to answer than it seems, because you seems, because you seems, because you look at it and think: oh, the look at it and think: oh, the look at it and think: oh, the customer was supposed to take one customer was supposed to take one , but he took two. Apparently, , but he took two. Apparently, , but he took two. Apparently, this is a customer error. this is a customer error. this is a customer error. But you're wrong, and But you're wrong, and But you're wrong, and I'll explain why in a I'll explain why in a I'll explain why in a minute. But first minute. But first , I'm Alex. I am an , I'm Alex. I am an , I'm Alex. I am an engine developer and architect at engine developer and architect at engine developer and architect at Hytale. Honestly, Hytale. Honestly, I spend most of my week I spend most of my week chasing chasing chasing crazy ideas and crazy ideas and crazy ideas and mistakes when you mistakes when you mistakes when you run into run into run into problems that problems that problems that don't make sense at all don't make sense at all don't make sense at all because you look because you look because you look at one side and at one side and at one side and think, "Oh, this should think, "Oh, this should think, "Oh, this should work." And then work." And then work." And then it turns out that it's it turns out that it's it turns out that it's not, and you just not, and you just not, and you just wasted wasted wasted three hours looking for three hours looking for three hours looking for one mistake. So one mistake. So one mistake. So when you read this when you read this when you read this stream of information, stream of information, stream of information, when the client and the server when the client and the server are different programs, are different programs, are different programs, when they're written in when they're written in when they're written in different languages, and the different languages, and the different languages, and the only thing that connects them only thing that connects them is the network is the network is the network protocol in between, protocol in between, protocol in between, where do you even start where do you even start ? And that's one of the ? And that's one of the ? And that's one of the biggest questions, and biggest questions, and biggest questions, and honestly, the one honestly, the one honestly, the one I I I like the most. So, like the most. So, like the most. So, based on just based on just based on just this example, I this example, I this example, I created a slide that created a slide that created a slide that shows you've seen

shows you've seen shows you've seen this on a client, this on a client, this on a client, right? Visual failure, right? Visual failure, right? Visual failure, incorrect incorrect incorrect quantity. quantity. quantity. The inventory The inventory The inventory displayed two displayed two displayed two instead of one. So, instead of one. So, instead of one. So, obviously, it's a obviously, it's a obviously, it's a client-side error. You client-side error. You client-side error. You open it, open it, open it, set set set breakpoints in the breakpoints in the breakpoints in the rendering code to rendering code to rendering code to see the state. You see the state. You see the state. You log the log the log the update cycle and update cycle and update cycle and waste an hour on it waste an hour on it waste an hour on it , because , because , because the error is actually not there. the error is actually not there. The fault is not with the client. So how do you find out where So how do you find out where the error actually is? And the error actually is? And the error actually is? And this is actually the main this is actually the main this is actually the main goal of this report: goal of this report: goal of this report: how to find the error without how to find the error without how to find the error without wasting time on wasting time on wasting time on each side of the each side of the each side of the problem? And the answer is problem? And the answer is in the middle. This is in the middle. This is in the middle. This is protocol. By protocol. By protocol. By comparing what comparing what comparing what each side thinks each side thinks each side thinks about the same thing, you about the same thing, you about the same thing, you can determine where the can determine where the can determine where the error lies. And this is the error lies. And this is the error lies. And this is the bisection point. It's a clear bisection point. It's a clear bisection point. It's a clear line that there line that there line that there 's an answer on both sides, but 's an answer on both sides, but 's an answer on both sides, but in the middle it in the middle it in the middle it should never should never should never change because you change because you change because you know what it says.

know what it says. So, you can So, you can ask ask ask each party, client each party, client each party, client and server, separately what they and server, separately what they and server, separately what they believe. And if they don't believe. And if they don't believe. And if they don't agree, your agree, your agree, your search space has already search space has already search space has already been halved, and been halved, and been halved, and you haven't opened a you haven't opened a you haven't opened a single file with single file with . And that's the whole method . And that's the whole method . And that's the whole method of breaking this of breaking this of breaking this problem down into parts. problem down into parts. If you fully If you fully trust the protocol trust the protocol trust the protocol or communication channel or communication channel or communication channel in the middle, you in the middle, you in the middle, you can clearly see who is telling can clearly see who is telling can clearly see who is telling the truth and who is lying. the truth and who is lying. So, this report should So, this report should give you four things. give you four things. The first is how The first is how to localize the error to localize the error . Which side of . Which side of . Which side of the channel is it on the channel is it on the channel is it on before before before you read you read any code? The second is how any code? The second is how any code? The second is how to run two to run two to run two debuggers at the same time debuggers at the same time so that it's not a so that it's not a so that it's not a horrible experience, horrible experience, horrible experience, but still gives you but still gives you but still gives you enough information enough information enough information to solve to solve to solve the problem. The third is how the problem. The third is how the problem. The third is how to read the channel data to read the channel data to read the channel data when both when both when both debuggers debuggers debuggers lie to you, because it lie to you, because it lie to you, because it happens, and how happens, and how happens, and how to determine this state to determine this state to determine this state when it happens.

when it happens. when it happens. Finally, Finally, Finally, failure patterns are specific failure patterns are specific failure patterns are specific to each to each to each runtime environment runtime environment . Your language has a . Your language has a . Your language has a list of problems on list of problems on list of problems on each side. And as long as each side. And as long as each side. And as long as you know how to you know how to you know how to identify identify identify errors in this language, you errors in this language, you errors in this language, you can quickly come can quickly come can quickly come to a preliminary to a preliminary to a preliminary guess just guess just guess just by looking at where the by looking at where the by looking at where the problem is. problem is. So, the first So, the first demonstration we'll demonstration we'll demonstration we'll look at brings look at brings look at brings us back to problem number us back to problem number us back to problem number one: where exactly one: where exactly one: where exactly is the error? is the error? is the error? We'll get back to We'll get back to We'll get back to that inventory error we that inventory error we talked about at the very talked about at the very beginning. beginning. The easiest way to The easiest way to do this is to briefly do this is to briefly do this is to briefly repeat what exactly repeat what exactly repeat what exactly happened. The player happened. The player happened. The player approaches the chest, approaches the chest, approaches the chest, and and and two two two items appear in the inventory. Although there should items appear in the inventory. Although there should items appear in the inventory. Although there should only be one. only be one. only be one. The chest should The chest should The chest should only give out one only give out one only give out one item. Usually the item. Usually the item. Usually the first desire will be: first desire will be: first desire will be: "Okay, let's go to "Okay, let's go to "Okay, let's go to the client and ask in the the client and ask in the the client and ask in the inventory code: what are you inventory code: what are you inventory code: what are you displaying?" "Well, displaying?" "Well, displaying?" "Well, the inventory shows the the inventory shows the the inventory shows the total number of total number of total number of records that were records that were records that were transferred to it." So if you transferred to it." So if you transferred to it." So if you apply a list, apply a list, apply a list, it just adds it just adds it just adds elements to that elements to that elements to that list. And now you're list. And now you're list. And now you're thinking, "Okay, thinking, "Okay, thinking, "Okay, so this came from the so this came from the so this came from the server." So now you server." So now you server." So now you move on to the move on to the move on to the server side. And you server side. And you server side. And you go back and forth between go back and forth between go back and forth between them for a long time. And although them for a long time. And although them for a long time. And although this is usually not a problem, it this is usually not a problem, it this is usually not a problem, it can be can be can be a lot of time consuming. So, we'll

a lot of time consuming. So, we'll a lot of time consuming. So, we'll now run the now run the now run the debugging process debugging process debugging process with a specific with a specific with a specific breakpoint to see breakpoint to see breakpoint to see what you would do when what you would do when what you would do when debugging on the debugging on the debugging on the client side. If client side. If client side. If we click "debug", we click "debug", it will work instantly. And suddenly we see And suddenly we see our inventory, where our inventory, where our inventory, where the number of records the number of records the number of records is zero. Because is zero. Because is zero. Because we haven't received we haven't received we haven't received any information yet. We'll any information yet. We'll any information yet. We'll move on to the move on to the move on to the next frame and next frame and next frame and let it let it let it run for a second. run for a second. And thanks to that, we And thanks to that, we now have this information now have this information that gave us our that gave us our that gave us our server and our server and our server and our inbox. Of course, you inbox. Of course, you inbox. Of course, you can't read the can't read the can't read the active thread there, active thread there, active thread there, but we can see on but we can see on but we can see on the client that it's not the client that it's not the client that it's not responding because it timed responding because it timed responding because it timed out.

out. However, as you However, as you follow this step by step, you follow this step by step, you follow this step by step, you see that a see that a see that a timeout occurs. Because the timeout occurs. Because the timeout occurs. Because the step-by-step step-by-step step-by-step debugging did not give debugging did not give debugging did not give us any us any us any information when information when information when establishing communication establishing communication establishing communication between the client and between the client and between the client and the server. A the server. A the server. A timeout occurs. And you timeout occurs. And you timeout occurs. And you go to the other go to the other go to the other side, and your state disappears side, and your state disappears , because , because , because the debugger the debugger the debugger only shows the information only shows the information only shows the information that is in a particular that is in a particular that is in a particular frame. And if you frame. And if you frame. And if you miss one frame, there's miss one frame, there's no going back. So no going back. So the question arises: how the question arises: how the question arises: how to find the place where to find the place where to find the place where the problem is hidden? the problem is hidden? The easiest way is to look at the is to look at the is to look at the problem itself. We problem itself. We problem itself. We were supposed to pick one were supposed to pick one were supposed to pick one item, but we were given item, but we were given item, but we were given two. The client two. The client two. The client displays two displays two displays two because in inventory, because in inventory, any number passed to it any number passed to it is returned as a is returned as a total. So, in total. So, in total. So, in our renderer, it our renderer, it our renderer, it returns whatever returns whatever returns whatever number it number it number it gets. This means gets. This means gets. This means that the problem is not the that the problem is not the that the problem is not the total number of total number of total number of items in the items in the items in the inventory. He was inventory. He was inventory. He was given this exact number.

given this exact number. Given this, our Given this, our problem actually problem actually problem actually lies on the server. lies on the server. And now you've found the And now you've found the part that really part that really part that really needs to be debugged needs to be debugged . And now you can . And now you can . And now you can dig into the dig into the dig into the source code to source code to source code to find the problem itself. find the problem itself. Normally you Normally you would do would do server-side debugging, but server-side debugging, but we already have a good we already have a good we already have a good idea where the idea where the idea where the problem lies. We problem lies. We problem lies. We know it will be in know it will be in know it will be in our inventory. And our inventory. And our inventory. And we know that this is most likely the we know that this is most likely the we know that this is most likely the logic logic logic that sends the that sends the that sends the information. So information. So information. So when we provide an when we provide an when we provide an item, you go item, you go item, you go through this code, and if through this code, and if through this code, and if you look at it, you look at it, you look at it, it outputs the item here it outputs the item here , and then sends , and then sends , and then sends the client an the client an the client an add packet. Now, on add packet. Now, on add packet. Now, on the other hand, your the other hand, your the other hand, your inventory has a inventory has a inventory has a confirmation where it confirmation where it confirmation where it expects a response from expects a response from expects a response from the customer: "Hey, I the customer: "Hey, I the customer: "Hey, I got this got this got this information." And then information." And then information." And then you have a you have a resubmission cycle because resubmission cycle because your inventory has to be your inventory has to be your inventory has to be correct. Now correct. Now correct. Now the question is, where does the question is, where does the problem actually lie? If the client the problem actually lie? If the client received two items received two items received two items and the server and the server and the server only reports one, how do you only reports one, how do you only reports one, how do you understand this? To do this, understand this? To do this, understand this? To do this, we will start our we will start our we will start our demo demo demo server, which is already server, which is already server, which is already in use. And in use. And in use. And if you run if you run if you run your server, you'll eventually your server, you'll eventually your server, you'll eventually run into a run into a run into a problem because

problem because problem because we won't be doing we won't be doing we won't be doing real- real- real- time debugging—that takes time debugging—that takes time debugging—that takes a lot of time. Our a lot of time. Our a lot of time. Our inventory sends a inventory sends a inventory sends a quantity, and that quantity, and that quantity, and that quantity has been quantity has been quantity has been set to set to set to one. But if we one. But if we one. But if we go to our go to our add send function, you'll add send function, you'll notice that we notice that we notice that we send the ID, subject, send the ID, subject, send the ID, subject, and quantity. This and quantity. This and quantity. This means we means we means we only send one, only send one, only send one, but the customer receives but the customer receives but the customer receives two. The hardest thing two. The hardest thing two. The hardest thing to notice about the client is that to notice about the client is that to notice about the client is that if you were to if you were to if you were to debug the client debug the client debug the client over and over again, you would over and over again, you would over and over again, you would realize that the second realize that the second realize that the second item is added item is added item is added a little later. And now a little later. And now a little later. And now you're completely you're completely you're completely confused because confused because confused because only only only one package is being sent. But this just one package is being sent. But this just one package is being sent. But this just points to a points to a points to a problem specifically in problem specifically in problem specifically in our our resend logic. Our resend logic. Our resend logic resend logic resend logic tells us tells us tells us that if the information is not that if the information is not that if the information is not received, then received, then received, then send it again send it again send it again because it because it because it was never acknowledged. was never acknowledged.

So if you look So if you look here at lines 41–44, you'll here at lines 41–44, you'll here at lines 41–44, you'll see a see a resend loop that resend loop that ensures that the packet ensures that the packet ensures that the packet actually reaches actually reaches actually reaches the client and there's no the client and there's no the client and there's no information leakage. And information leakage. And information leakage. And he only checks it he only checks it he only checks it after sending. Which, in after sending. Which, in after sending. Which, in fact, is the problem. fact, is the problem. fact, is the problem. So, you So, you only acknowledge receipt only acknowledge receipt after the data has been after the data has been after the data has been resent. That's the resent. That's the whole problem. Simply whole problem. Simply by determining that the client was by determining that the client was by determining that the client was only re- only re- only re- reading and reading and reading and displaying the existing displaying the existing displaying the existing information, you can information, you can information, you can understand that the understand that the understand that the problem is on the problem is on the problem is on the server side. And when you server side. And when you server side. And when you know that the problem is know that the problem is know that the problem is on the server, you can on the server, you can on the server, you can find the exact find the exact find the exact piece of logic where piece of logic where piece of logic where it occurs and it occurs and it occurs and come across such come across such come across such little things as little things as little things as confirmation that confirmation that confirmation that was read and was read and was read and checked checked checked after sending. And that's the after sending. And that's the after sending. And that's the whole problem. This is the whole problem. This is the whole problem. This is the essence of this essence of this essence of this particular method.

particular method. particular method. You ask each You ask each You ask each side what they “ side what they “ think” and then think” and then think” and then find out exactly where the find out exactly where the find out exactly where the problem lies. problem lies. The client thinks it's The client thinks it's a two, the server thinks it's a two, the server thinks it's a one, and you a one, and you a one, and you realize that the client realize that the client realize that the client lied, even though lied, even though lied, even though in reality he told the in reality he told the in reality he told the truth. And this is where the truth. And this is where the complexity of this complexity of this problem begins. problem begins. When looking at all these When looking at all these errors, you errors, you errors, you come across a come across a come across a lot of them. And lot of them. And lot of them. And in the end it all in the end it all in the end it all comes down to the point of comes down to the point of comes down to the point of bisection. The client bisection. The client bisection. The client thinks one thing about the thinks one thing about the thinks one thing about the entity, and the server thinks entity, and the server thinks another, and you rush another, and you rush another, and you rush to blame the to blame the to blame the communication channel. The process communication channel. The process communication channel. The process I'm trying to I'm trying to I'm trying to show you proves show you proves show you proves that blaming that blaming that blaming the channel is a mistake. The channel the channel is a mistake. The channel is the truth. Therefore, it is the truth. Therefore, it is the truth. Therefore, it must be assumed must be assumed must be assumed that both the client and the server are that both the client and the server are that both the client and the server are lying. And if you lying. And if you lying. And if you believe they are believe they are believe they are both lying, how can you both lying, how can you both lying, how can you prove prove prove whose side is the problem?

whose side is the problem? whose side is the problem? And I think And I think I showed at least some of that in the I showed at least some of that in the first demo. first demo. Now let's move on to the Now let's move on to the second example, where second example, where second example, where we'll talk about how to we'll talk about how to we'll talk about how to use two use two use two debuggers at the same time debuggers at the same time . Honestly, this is . Honestly, this is . Honestly, this is not an easy task. not an easy task. So let's So let's discuss how this discuss how this discuss how this can be implemented. can be implemented. The first step is to see what to see what debuggers are actually capable of. And debuggers are actually capable of. And the answer to this the answer to this the answer to this question is—what to do question is—what to do question is—what to do when the error when the error when the error appears on only appears on only appears on only one side? And this one side? And this one side? And this raises questions about the raises questions about the raises questions about the protocol design itself. protocol design itself. This is easy This is easy to demonstrate. to demonstrate. If you notice, If you notice, the client is a little " the client is a little " buggy". And no, it's not buggy". And no, it's not buggy". And no, it's not your screen. This is a real your screen. This is a real your screen. This is a real problem. It takes a problem. It takes a problem. It takes a step back almost step back almost step back almost every beat of the loop, every beat of the loop, every beat of the loop, which is an obvious " which is an obvious " stutter glitch". So stutter glitch". So , how do you detect the stutter failure itself , how do you detect the stutter failure itself when you need to debug both debug both sides of the protocol simultaneously?

sides of the protocol simultaneously? The answer is to The answer is to make your make your make your code more code more code more readable over the network. readable over the network. And it's surprisingly easy to And it's surprisingly easy to do. The do. The simplest approach is to start with the thought, “Oh, I’ll is to start with the thought, “Oh, I’ll is to start with the thought, “Oh, I’ll just see in just see in just see in what order what order what order each each each packet is received, set a packet is received, set a packet is received, set a breakpoint, and when it breakpoint, and when it breakpoint, and when it fires, I’ll check fires, I’ll check fires, I’ll check that side and get the that side and get the that side and get the information I need information I need .” Well, we .” Well, we .” Well, we proved earlier that proved earlier that proved earlier that breakpoints on the client breakpoints on the client breakpoints on the client cause a crash, a cause a crash, a cause a crash, a timeout, timeout, timeout, because because because time runs out. time runs out. time runs out. So, now you have a So, now you have a So, now you have a headache. How do headache. How do headache. How do I read I read I read information from the client information from the client information from the client if a timeout occurs if a timeout occurs because the server is because the server is because the server is no longer communicating with it no longer communicating with it no longer communicating with it through breakpoints on through breakpoints on through breakpoints on both sides? And when both sides? And when both sides? And when your your heartbeats aren't heartbeats aren't heartbeats aren't being sent, you being sent, you being sent, you disconnect. This is a disconnect. This is a disconnect. This is a logical answer to logical answer to logical answer to this problem. So I this problem. So I this problem. So I want to draw your want to draw your want to draw your attention to a few protocol changes you attention to a few protocol changes you can make to make your can make to make your life easier. The life easier. The first one is first one is here. This is here. This is here. This is the header that I the header that I the header that I record for each record for each record for each individual packet individual packet individual packet sent over the sent over the sent over the network to the client.

network to the client. The most important two are right here, on lines 33 and 34 here, on lines 33 and 34 , where we send the , where we send the , where we send the game tick and game tick and sequence value. Now, this can be converted this can be converted into what is called a into what is called a into what is called a correlation ID. The correlation ID. The correlation ID. The correlation ID tracks correlation ID tracks correlation ID tracks both the both the both the sending sending sending and and and receiving side of packets, so you receiving side of packets, so you receiving side of packets, so you know on which tick know on which tick know on which tick they were sent they were sent they were sent or received and what or received and what or received and what their sequence their sequence their sequence number is. This means number is. This means number is. This means you make sure you you make sure you you make sure you are looking at the are looking at the are looking at the right package every time. right package every time. That way, you won't have the hassle of looking at hassle of looking at two different packages two different packages two different packages that arrived that arrived that arrived late and you won't be late and you won't be late and you won't be saying, "Oh, this one is saying, "Oh, this one is saying, "Oh, this one is perfect." perfect." Because what happens Because what happens if one comes if one comes if one comes one tick later? In one tick later? In one tick later? In such a situation, you such a situation, you such a situation, you encounter this encounter this encounter this jitter glitch, which is the jitter glitch, which is the jitter glitch, which is the crux of the problem. crux of the problem.

Moving in a state Moving in a state where the server is the master, where the server is the master, where the server is the master, as in this example, is as in this example, is as in this example, is inherently dependent inherently dependent inherently dependent on ticks. This means on ticks. This means on ticks. This means that every time that every time that every time you consider this you consider this you consider this problem, there is a cycle of problem, there is a cycle of problem, there is a cycle of reconciliation. And reconciliation. And reconciliation. And the easiest way to the easiest way to the easiest way to show this is to go show this is to go show this is to go to the client and to the client and to the client and look at the same look at the same look at the same part. On the client, part. On the client, part. On the client, when you read when you read when you read the header, you the header, you the header, you read your thread and read your thread and read your thread and your sequence. your sequence. This is made easier by the fact This is made easier by the fact that Rider that Rider that Rider by default by default by default shows us what exactly shows us what exactly shows us what exactly it is reading into the it is reading into the it is reading into the header values ​​on both header values ​​on both header values ​​on both sides, so you can sides, so you can sides, so you can quickly and easily quickly and easily quickly and easily identify what identify what identify what values ​​you are looking for. values ​​you are looking for. One of the One of the most powerful most powerful most powerful features of features of features of debuggers is debuggers is debuggers is the ability to read your the ability to read your the ability to read your stack, current memory, stack, current memory, stack, current memory, iterations, and all iterations, and all iterations, and all the information that the information that the information that the client or server is the client or server is the client or server is actively reading at actively reading at actively reading at that moment. that moment.

The most interesting thing here is that The most interesting thing here is that now, having the now, having the correlation ID, we correlation ID, we know exactly which know exactly which know exactly which packet the packet the packet the problem comes with. In this problem comes with. In this problem comes with. In this simple demonstration, simple demonstration, simple demonstration, you see "stutters"—a you see "stutters"—a sign that something is sign that something is sign that something is causing the client to causing the client to causing the client to constantly jump constantly jump constantly jump back and forth from back and forth from back and forth from their position. The their position. The easiest easiest solution, again solution, again solution, again , would be to , would be to , would be to blame everything on the blame everything on the blame everything on the client. In this client. In this client. In this situation, you would even be situation, you would even be situation, you would even be right, but you would be right, but you would be right, but you would be pointing out the wrong pointing out the wrong pointing out the wrong problem. You will start problem. You will start problem. You will start to study the code of motion and to study the code of motion and to study the code of motion and ask, "Why does motion ask, "Why does motion ask, "Why does motion twitch?" And when twitch?" And when twitch?" And when the motion the motion the motion returns the returns the returns the correct value every time, correct value every time, correct value every time, you'll ask you'll ask you'll ask the renderer, "Why are you the renderer, "Why are you the renderer, "Why are you updating so updating so updating so slowly?" And that's exactly the slowly?" And that's exactly the complexity of this complexity of this particular bug. particular bug. Tick ​​skew occurs Tick ​​skew occurs when a response from when a response from when a response from the server arrives, the server arrives, the server arrives, relatively speaking, relatively speaking, relatively speaking, late, although late, although late, although technically it technically it technically it arrives exactly when it arrives exactly when it arrives exactly when it should. You could should. You could should. You could set a trap to set a trap to set a trap to prove this, but there's an prove this, but there's an prove this, but there's an easier way: as easier way: as easier way: as we proved in the last we proved in the last we proved in the last demonstration, you can demonstration, you can demonstration, you can set conditional set conditional set conditional breakpoints on both breakpoints on both breakpoints on both sides, which is very sides, which is very sides, which is very useful. Conditional

useful. Conditional useful. Conditional breakpoints in Rider breakpoints in Rider breakpoints in Rider work even better, work even better, work even better, because we can set because we can set because we can set the condition: snapshot.tick the condition: snapshot.tick the condition: snapshot.tick equals 400. This gives equals 400. This gives equals 400. This gives us one specific us one specific us one specific tick to analyze. We tick to analyze. We tick to analyze. We will consider the 400th tick on the will consider the 400th tick on the will consider the 400th tick on the client side. client side. Then we'll do the same Then we'll do the same on the server side, on the server side, on the server side, right? And this is where right? And this is where right? And this is where Rider and CLion's Rider and CLion's Rider and CLion's cross-platform cross-platform cross-platform debugging capabilities become debugging capabilities become debugging capabilities become so powerful. so powerful. By establishing a snapshot of By establishing a snapshot of the situation on both sides of the situation on both sides of the situation on both sides of the problem, you can the problem, you can the problem, you can quickly and easily quickly and easily quickly and easily see where it is see where it is see where it is coming from. So, we have a coming from. So, we have a simulation running on our server. This really simulation running on our server. This really helps us helps us helps us pinpoint the exact pinpoint the exact pinpoint the exact place in the simulation where place in the simulation where the matching is being performed, namely the matching is being performed, namely inside our inside our inside our run tick method.

run tick method. Looking at our code, Looking at our code, you would of course understand you would of course understand you would of course understand it better if you it better if you it better if you wrote it yourself, but wrote it yourself, but wrote it yourself, but we will set a we will set a we will set a breakpoint at line 135 breakpoint at line 135 breakpoint at line 135 and, again, and, again, and, again, add a condition that tick add a condition that tick add a condition that tick is equal to 400. This is the is equal to 400. This is the is equal to 400. This is the power of power of power of cross-platform cross-platform cross-platform debugging in CLion and debugging in CLion and debugging in CLion and Rider. If you run Rider. If you run Rider. If you run this in this in this in debug mode, you debug mode, you debug mode, you will get a state where will get a state where will get a state where both ticks both ticks both ticks are equal to 400. are equal to 400. The problem is with the The problem is with the same same same heartbeat signal we heartbeat signal we heartbeat signal we talked about earlier. talked about earlier. Of course, for most Of course, for most people, the solution will people, the solution will people, the solution will simply be to not require simply be to not require simply be to not require heartbeat in the heartbeat in the heartbeat in the development environment. You don't development environment. You don't development environment. You don't need him. So, need him. So, need him. So, the client is running at the client is running at the client is running at 20 hertz, or 20 hertz, or 20 hertz, or 20 ticks per second, and 20 ticks per second, and 20 ticks per second, and after 3 seconds it times after 3 seconds it times after 3 seconds it times out out out because it did not because it did not because it did not receive a heartbeat signal receive a heartbeat signal receive a heartbeat signal from the server. And now from the server. And now from the server. And now we need to understand we need to understand we need to understand how to get around this problem how to get around this problem how to get around this problem when designing? The when designing? The easiest way easiest way to get around this problem to get around this problem is another trick that is another trick that debuggers give us: debuggers give us: to control exactly to control exactly to control exactly how your debugger how your debugger how your debugger stops.

stops. The easiest way The easiest way to do this is to do this is to set the to set the suspend execution mode to "none". This way your This way your server continues to server continues to server continues to run run run continuously, you continuously, you continuously, you still get still get still get breakpoint information breakpoint information breakpoint information , but you don't , but you don't , but you don't get a heartbeat failure get a heartbeat failure get a heartbeat failure that prevents the client from that prevents the client from that prevents the client from receiving the data it receiving the data it receiving the data it needs to needs to needs to continue working. continue working. Another great Another great feature of feature of feature of debugging on this debugging on this debugging on this platform is platform is platform is that if you are that if you are that if you are having trouble with a having trouble with a having trouble with a particular particular particular debugger, on the debugger, on the debugger, on the Rider side you can Rider side you can Rider side you can even tell it even tell it even tell it to create a stack dump to create a stack dump to create a stack dump when it happens when it happens when it happens to get all to get all to get all the information about where the the information about where the breakpoint was triggered and save breakpoint was triggered and save it for later. This also it for later. This also it for later. This also applies to CLion, but applies to CLion, but applies to CLion, but when when when debugging natively it debugging natively it debugging natively it sometimes works with sometimes works with sometimes works with errors. So I errors. So I errors. So I advise you to just advise you to just advise you to just use this use this use this suspension system suspension system suspension system . Now, another . Now, another . Now, another very clever very clever very clever way to get around this is way to get around this is to use streams to use streams . If you are building a . If you are building a . If you are building a system using

system using threads, you threads, you know right away that you are going to know right away that you are going to know right away that you are going to suspend a thread. And suspend a thread. And suspend a thread. And I actually I actually I actually anticipated this in the anticipated this in the anticipated this in the design. In the simulation, design. In the simulation, design. In the simulation, we run the entire we run the entire we run the entire process in a single process in a single process in a single thread. So the simulation thread. So the simulation thread. So the simulation lives in its own lives in its own lives in its own thread, separate from the thread, separate from the thread, separate from the main and main and main and network threads, network threads, network threads, which would be which would be which would be suspended. This suspended. This suspended. This means that a heartbeat means that a heartbeat means that a heartbeat running in a loop on the running in a loop on the running in a loop on the network will never network will never network will never stop stop stop sending, sending, sending, as you can as you can as you can pause a pause a pause a specific stream in both specific stream in both specific stream in both CLion and Rider—the CLion and Rider—the CLion and Rider—the only two only two only two tools I'm tools I'm tools I'm showing today. showing today. However, you can However, you can do this on other do this on other do this on other platforms as well, where platforms as well, where platforms as well, where the terminology may the terminology may the terminology may only only only differ slightly. One of the differ slightly. One of the differ slightly. One of the really cool things about really cool things about really cool things about being able being able being able to control to control to control parts of your parts of your parts of your debugger is debugger is debugger is also being able to also being able to also being able to control conditions control conditions control conditions at runtime. at runtime.

If you notice If you notice a condition in a particular tick, a condition in a particular tick, a condition in a particular tick, say tick 193 say tick 193 say tick 193 coming to the client, coming to the client, coming to the client, but the server but the server but the server only sent 191, then you know there only sent 191, then you know there only sent 191, then you know there is a problem on the is a problem on the is a problem on the server side where the server side where the server side where the data is being sent out of data is being sent out of order, and then you order, and then you can look for this can look for this can look for this error a little error a little error a little differently. So, in this differently. So, in this differently. So, in this situation, it is situation, it is situation, it is easy to find the error. easy to find the error. Since we know Since we know about the about the about the synchronization cycle, synchronization cycle, synchronization cycle, the server will send the server will send the server will send information with a information with a information with a delay of one clock cycle delay of one clock cycle . And with . And with . And with the debugger for the debugger for the debugger for each thread, you each thread, you each thread, you can quickly and can quickly and can quickly and easily conclude easily conclude easily conclude that the problem is the that the problem is the that the problem is the clock delay. So, clock delay. So, clock delay. So, how do we fix this how do we fix this how do we fix this problem? I'm not here to problem? I'm not here to problem? I'm not here to answer that answer that answer that . I'm here to . I'm here to . I'm here to answer the answer the answer the question of how to find the question of how to find the question of how to find the problem. Of course, there are problem. Of course, there are many ways to fix this, and many ways to fix this, and there are many people who are more there are many people who are more there are many people who are more knowledgeable about networking than knowledgeable about networking than knowledgeable about networking than I am, so I'll I am, so I'll I am, so I'll leave it to them. But if leave it to them. But if leave it to them. But if you can find you can find you can find the problem, you can always the problem, you can always the problem, you can always find the solution find the solution find the solution if you know where it is.

if you know where it is. And if you can And if you can find the problem in 10 find the problem in 10 find the problem in 10 minutes, as I just minutes, as I just minutes, as I just told you, it's not told you, it's not told you, it's not hours of switching hours of switching hours of switching between different between different between different parts of the source parts of the source parts of the source code until you code until you code until you find where the find where the find where the problem actually is. problem actually is. So, we've talked a lot So, we've talked a lot about how you about how you about how you find a problem and find a problem and find a problem and how you use the how you use the how you use the debugger to debugger to debugger to find it. But how do you find it. But how do you find it. But how do you find the problem find the problem when it's actually in the when it's actually in the communication channel itself communication channel itself ? And this is a very complex ? And this is a very complex ? And this is a very complex problem. So we'll problem. So we'll talk about that a little bit. In our talk about that a little bit. In our previous bugs, previous bugs, we talked about how you we talked about how you can track down can track down can track down any specific any specific any specific problem using the problem using the debugger. But how do debugger. But how do you track wolves you track wolves you track wolves that act like that act like that act like chests and a chest that chests and a chest that chests and a chest that acts like a wolf?

acts like a wolf? Well, it looks like a Well, it looks like a data problem. And data problem. And data problem. And again, we'll talk again, we'll talk again, we'll talk a little bit about the design philosophy of a little bit about the design philosophy of fixing both fixing both sides of the problem. In sides of the problem. In sides of the problem. In this situation, you are again this situation, you are again this situation, you are again quick quick quick to blame the client to blame the client to blame the client because the client because the client because the client is clearly showing that is clearly showing that is clearly showing that the chest is a wolf and the two the chest is a wolf and the two the chest is a wolf and the two wolves are chests. And wolves are chests. And wolves are chests. And being able to quickly being able to quickly being able to quickly identify this identify this identify this problem points you problem points you problem points you to the customer. So, what's to the customer. So, what's to the customer. So, what's going on? We going on? We going on? We go in and go in and go in and open the client. open the client. We look at our We look at our information and information and information and go to our go to our go to our transfer code. And we transfer code. And we transfer code. And we know that our channel is know that our channel is know that our channel is reading the information reading the information reading the information correctly. So we correctly. So we correctly. So we go look at our go look at our go look at our protocol. And our protocol. And our protocol. And our protocol defines the protocol defines the protocol defines the entity type in entity type in entity type in number format. So, number format. So, number format. So, we have a player, a box, a we have a player, a box, a we have a player, a box, a wolf, a chest, and a torch.

wolf, a chest, and a torch. wolf, a chest, and a torch. And they all look And they all look And they all look right. So right. So right. So now that we are now that we are now that we are getting the correct getting the correct getting the correct information, it looks information, it looks information, it looks like the server like the server like the server is sending the wrong is sending the wrong is sending the wrong information because the information because the information because the entity types are entity types are entity types are correct. So why is correct. So why is correct. So why is the server sending the the server sending the the server sending the wrong wrong wrong entity type? So, if entity type? So, if entity type? So, if we go back to we go back to we go back to our server, our server, our server, the question arises: the question arises: the question arises: what is the problem here? what is the problem here? So, in our So, in our setup loop, we setup loop, we setup loop, we prepare the data and then prepare the data and then prepare the data and then feed it into our feed it into our feed it into our simulation. OK. simulation. OK. simulation. OK. The problem must be in The problem must be in The problem must be in our simulation. So our simulation. So , our simulation , our simulation , our simulation creates a box, a wolf, a creates a box, a wolf, a creates a box, a wolf, a chest, and a wolf. Our chest, and a wolf. Our chest, and a wolf. Our two wolves two wolves two wolves are configured are configured are configured correctly, as is the box, correctly, as is the box, correctly, as is the box, but they are but they are but they are displayed displayed displayed backwards, even though the server backwards, even though the server backwards, even though the server is sending the correct is sending the correct is sending the correct information. Now that information. Now that information. Now that we know that both we know that both we know that both the server and the client the server and the client the server and the client are sending data that they are sending data that they are sending data that they believe is correct believe is correct , we can ask , we can ask , we can ask the question: Okay, where the question: Okay, where the question: Okay, where exactly is the error in the exactly is the error in the exactly is the error in the data transmission? And to data transmission? And to data transmission? And to find out, the find out, the find out, the easiest way to first easiest way to first easiest way to first look at the

look at the look at the protocol itself. Our protocol itself. Our protocol itself. Our protocol is generated in a protocol is generated in a protocol is generated in a rather strange way, rather strange way, rather strange way, as it is supposed to be as it is supposed to be as it is supposed to be bilingual. We should bilingual. We should bilingual. We should have one side in C# and the have one side in C# and the have one side in C# and the other in C++. other in C++. Obviously, you Obviously, you generate this generate this generate this information when you information when you information when you need it. And, need it. And, need it. And, in fact, it is not in fact, it is not in fact, it is not needed. You don't needed. You don't needed. You don't need C++ on the C# side, need C++ on the C# side, need C++ on the C# side, nor C# on the C++ side. So, nor C# on the C++ side. So, nor C# on the C++ side. So, we're back to we're back to we're back to another design another design another design decision: if decision: if decision: if you look at the you look at the you look at the protocol, there's a protocol, there's a protocol, there's a player, a box, a wolf, a player, a box, a wolf, a player, a box, a wolf, a chest, a torch, chest, a torch, chest, a torch, right? And on the other side: right? And on the other side: right? And on the other side: player, box, wolf, player, box, wolf, player, box, wolf, chest, torch. This chest, torch. This chest, torch. This looks absolutely looks absolutely looks absolutely correct. The problem is correct. The problem is correct. The problem is that the generated that the generated that the generated files we build files we build files we build based on this for C based on this for C based on this for C ++ contain an error.

++ contain an error. Our C++ side sends Our C++ side sends this information a little this information a little this information a little backwards. So backwards. So backwards. So our C++ protocol our C++ protocol our C++ protocol shows a box and then shows a box and then shows a box and then a wolf. Yes? But on a wolf. Yes? But on a wolf. Yes? But on our side, C#, which our side, C#, which our side, C#, which I believe is currently I believe is currently I believe is currently being overridden. Um, oh, being overridden. Um, oh, being overridden. Um, oh, there it is. Here we are there it is. Here we are there it is. Here we are . It . It . It appears as a appears as a appears as a wolf and then a chest. wolf and then a chest. wolf and then a chest. Of course, I had to Of course, I had to Of course, I had to create it in two create it in two create it in two different ways. different ways. different ways. Therefore, there is an outdated Therefore, there is an outdated Therefore, there is an outdated version because we have not version because we have not version because we have not updated the client code. updated the client code. But you'll notice that But you'll notice that here it's a wolf, then here it's a wolf, then here it's a wolf, then a chest, and here it's a box, a chest, and here it's a box, a chest, and here it's a box, then a wolf, because we then a wolf, because we then a wolf, because we added the "box" type on added the "box" type on added the "box" type on the server, but didn't the server, but didn't the server, but didn't update the client to update the client to update the client to adapt to adapt to adapt to it. And this is the problem of it. And this is the problem of it. And this is the problem of protocol generation.

protocol generation. There are many solutions There are many solutions to this particular to this particular to this particular problem. And one of the problem. And one of the problem. And one of the best is best is to generate both to generate both to generate both sides of the communication sides of the communication sides of the communication using using using a single a single a single centralized centralized centralized protocol type, and then protocol type, and then protocol type, and then simply generate simply generate simply generate the code for the desired the code for the desired the code for the desired language. There are language. There are language. There are many tools for this. many tools for this. many tools for this. For example, Protobuf. I For example, Protobuf. I For example, Protobuf. I know this can be know this can be know this can be done in C#. You done in C#. You done in C#. You can write your can write your can write your protocol in a protocol in a protocol in a programming language and programming language and programming language and then generate then generate then generate binding types as binding types as binding types as needed. Or it can be needed. Or it can be needed. Or it can be done in C++. You done in C++. You done in C++. You just decide that just decide that just decide that one side should be one side should be one side should be the answer. The other the answer. The other the answer. The other party should party should listen to this response. If listen to this response. If you are in a hurry every time, you are in a hurry every time, you are in a hurry every time, you are faced with you are faced with you are faced with the question: how to solve the question: how to solve the question: how to solve a problem when it is a problem when it is a problem when it is required in two required in two required in two languages? Then you languages? Then you languages? Then you write bindings and write bindings and write bindings and create new create new create new elements. Then you elements. Then you elements. Then you choose the side that choose the side that choose the side that will be true. This will be true. This will be true. This exposes the problem exposes the problem exposes the problem directly, directly, directly, as your client as your client as your client now thinks your now thinks your now thinks your wolves are chests. This is wolves are chests. This is wolves are chests. This is because the server because the server because the server thinks it is sending the thinks it is sending the thinks it is sending the correct

correct correct enumeration value, but two is enumeration value, but two is enumeration value, but two is a chest for our a chest for our a chest for our client, and a wolf is one. client, and a wolf is one. So, our chest So, our chest turns into a turns into a turns into a torch. This torch. This torch. This turns into a turns into a turns into a number four because it's number four because it's number four because it's not even not even not even supported. This supported. This supported. This list does not exist. This is another list does not exist. This is another list does not exist. This is another very quick very quick very quick way to notice something that doesn't way to notice something that doesn't way to notice something that doesn't look look look right. This right. This right. This indicates a problem quite quickly indicates a problem quite quickly indicates a problem quite quickly . I . I . I figured out this whole figured out this whole figured out this whole problem in 5 minutes. problem in 5 minutes. Instead of Instead of wasting time on: " wasting time on: " Yes, the server is sending Yes, the server is sending Yes, the server is sending correctly. The client correctly. The client correctly. The client is sending correctly. is sending correctly. is sending correctly. I need I need I need to check the connection. to check the connection. to check the connection. That's the problem." That's the problem." That's the problem." This is a really quick This is a really quick This is a really quick way to do this— way to do this— go back to the go back to the go back to the bisection method again. bisection method again. bisection method again. See what they " See what they " think" about the object. think" about the object. think" about the object. Both sides Both sides Both sides believe they have the believe they have the believe they have the correct information correct information , and both get the , and both get the , and both get the correct meaning of correct meaning of correct meaning of the list. Both the list. Both the list. Both get the number two get the number two get the number two or the number one.

or the number one. or the number one. They just consider They just consider They just consider them different, and that's when them different, and that's when them different, and that's when your protocol your protocol your protocol changed. And when your changed. And when your changed. And when your protocol protocol protocol is wrong, you is wrong, you is wrong, you can review can review can review it and find the it and find the it and find the problem. But problem. But problem. But pay attention: pay attention: pay attention: the last thing I did the last thing I did the last thing I did was check was check was check the protocol and look for a the protocol and look for a the protocol and look for a problem in it. This problem in it. This problem in it. This brings us back to the brings us back to the brings us back to the original method original method original method I was talking about. I was talking about. If you If you check the protocol first check the protocol first , you waste , you waste , you waste a lot of time. If you a lot of time. If you a lot of time. If you check check check one side first, you're one side first, you're one side first, you're wasting a lot of wasting a lot of wasting a lot of time too. The method is to time too. The method is to time too. The method is to find out find out find out what both what both what both sides think before sides think before sides think before testing one of them testing one of them . Now there is a second . Now there is a second . Now there is a second way this way this way this happens. And it is happens. And it is happens. And it is very difficult very difficult very difficult to actually detect it. So when to actually detect it. So when to actually detect it. So when you have a problem you have a problem you have a problem that looks like this, that looks like this, where everything is just where everything is just stuck at the top and stuck at the top and stuck at the top and nothing nothing nothing happens. You happens. You happens. You glitch a little for glitch a little for glitch a little for a second and then a second and then a second and then you're back to square one you're back to square one you're back to square one . This is a really . This is a really . This is a really strange problem because strange problem because strange problem because once again it seems once again it seems once again it seems as if the protocol is lying.

as if the protocol is lying. It's hard to It's hard to pinpoint this pinpoint this pinpoint this problem because problem because problem because the server is obviously the server is obviously the server is obviously sending information, sending information, sending information, as objects as objects as objects have appeared on the screen. have appeared on the screen. The client is obviously The client is obviously displaying things, but displaying things, but displaying things, but they are not arriving the way they are not arriving the way the server was the server was the server was instructed to send them. instructed to send them. And in this situation, you are And in this situation, you are forced to look at the forced to look at the forced to look at the protocol. And this is a protocol. And this is a protocol. And this is a really big really big really big mistake that I want to mistake that I want to mistake that I want to address here—when you address here—when you address here—when you don't look at the don't look at the don't look at the protocol. So, protocol. So, protocol. So, let's talk let's talk let's talk about this for at least a minute. The about this for at least a minute. The entities entities are indeed in are indeed in are indeed in impossible places, and impossible places, and impossible places, and it's hard to show without it's hard to show without it's hard to show without some debugging some debugging some debugging on the client itself, on the client itself, on the client itself, but it's all a bit but it's all a bit but it's all a bit confusing. Everything is a confusing. Everything is a confusing. Everything is a little off little off . So, how to . So, how to . So, how to fix this? fix this? The easiest way The easiest way to fix this is to to fix this is to first look at first look at first look at what each side is sending what each side is sending what each side is sending . So first . So first . So first we'll move on to our we'll move on to our we'll move on to our simulation. And our simulation. And our simulation. And our simulation is responsible simulation is responsible simulation is responsible for all the information that is for all the information that is for all the information that is transmitted. And to transmitted. And to transmitted. And to do that, we're do that, we're do that, we're going to go very going to go very going to go very quickly. Yes. You know, quickly. Yes. You know, quickly. Yes. You know, every time I every time I every time I did a live did a live did a live demo, it demo, it demo, it worked flawlessly

worked flawlessly . Today—no. So I'll do my . Today—no. So I'll do my . Today—no. So I'll do my best to best to best to just just just talk about it, I talk about it, I talk about it, I apologize to apologize to apologize to everyone. Ahem. When you everyone. Ahem. When you everyone. Ahem. When you look at look at look at the information you the information you the information you send through a send through a send through a protocol, obviously protocol, obviously protocol, obviously your protocol is the your protocol is the your protocol is the source of truth. It source of truth. It source of truth. It contains what contains what contains what needs to be sent, needs to be sent, needs to be sent, the types in which it the types in which it the types in which it needs to be sent, and needs to be sent, and needs to be sent, and exactly how it should be exactly how it should be exactly how it should be done in the protocol. done in the protocol. It also contains a It also contains a list of messages list of messages list of messages you send. you send. you send. So, I don't see So, I don't see So, I don't see any problems here. Everything any problems here. Everything any problems here. Everything is being sent is being sent is being sent correctly and there correctly and there correctly and there are no problems. In are no problems. In are no problems. In our simulation, everything our simulation, everything our simulation, everything is sent, we is sent, we is sent, we write our write our write our header, we header, we header, we send our send our send our header with the header with the header with the correct correct correct information, we information, we information, we send our send our send our entity with the ID, the type, entity with the ID, the type, entity with the ID, the type, our X and Y, the our X and Y, the our X and Y, the health. So we health. So we health. So we just send just send just send our X and Y values. our X and Y values. our X and Y values. So the wired So the wired So the wired entity entity entity is sent is sent is sent correctly. And that's correctly. And that's correctly. And that's when you want when you want when you want to look at the client to look at the client to look at the client because, well, our X and Y because, well, our X and Y because, well, our X and Y are being sent are being sent are being sent correctly, so where is the correctly, so where is the problem really? So, if problem really? So, if we move to we move to we move to the client and to our the client and to our the client and to our program. In fact, program. In fact, program. In fact, it will be in our it will be in our it will be in our network code. In network code. In network code. In our receiving, we our receiving, we our receiving, we will receive

will receive will receive information and information and information and read it to read it to read it to have data about what is have data about what is have data about what is happening. And the happening. And the happening. And the easiest way easiest way easiest way to look at it is to look at it is to look at it is our applied our applied our applied essence. When we essence. When we essence. When we apply an apply an apply an entity, we take entity, we take entity, we take the values ​​of X and Y and the values ​​of X and Y and the values ​​of X and Y and consider them consider them consider them simply. So, to simply. So, to simply. So, to do this, let's do this, let's do this, let's run, uh, console.write run, uh, console.write . Let's make a write . Let's make a write . Let's make a write line so it doesn't line so it doesn't line so it doesn't look terrible. And look terrible. And look terrible. And let's output all let's output all let's output all the values ​​we the values ​​we the values ​​we get from get from get from the server. Now, if the server. Now, if the server. Now, if we run the client, we run the client, you'll notice that these you'll notice that these numbers are meaningless. numbers are meaningless. numbers are meaningless. We have 3.6*10 ^ -41, uh, -13 We have 3.6*10 ^ -41, uh, -13 We have 3.6*10 ^ -41, uh, -13 billion, like... These billion, like... These billion, like... These numbers are absurd. And numbers are absurd. And numbers are absurd. And when the numbers are so when the numbers are so when the numbers are so wrong, there is a wrong, there is a wrong, there is a problem, and that problem, and that problem, and that problem is problem is problem is hidden somewhere. So, hidden somewhere. So, hidden somewhere. So, obviously, we've already obviously, we've already obviously, we've already identified the problem.

identified the problem. The server sends its The server sends its X and Y values, and X and Y values, and X and Y values, and the client reads them, and the client reads them, and the client reads them, and the client reads the client reads the client reads absurd values. absurd values. So let's So let's log what the log what the log what the server side says. When server side says. When server side says. When you log them, the server you log them, the server you log them, the server will show that will show that will show that it is actually sending the it is actually sending the it is actually sending the correct X and Y. correct X and Y. correct X and Y. Of course, I can't Of course, I can't Of course, I can't restart the server restart the server because something because something because something has broken somewhere, but it has broken somewhere, but it has broken somewhere, but it will show the correct will show the correct will show the correct X and Y X and Y X and Y because to it, X and Y are because to it, X and Y are because to it, X and Y are correct. So, where is the correct. So, where is the correct. So, where is the problem? Well, problem? Well, problem? Well, the problem is that on the problem is that on the problem is that on this side of our this side of our this side of our write loop, we are write loop, we are write loop, we are actually writing actually writing actually writing padding values padding values padding values to make sure that to make sure that to make sure that the information we are the information we are the information we are sending is sending is sending is padded. And it is this padded. And it is this padded. And it is this padding value that padding value that padding value that causes the problem.

causes the problem. The packet should be 106 The packet should be 106 bytes. So, it should bytes. So, it should bytes. So, it should send your type, send your type, send your type, your ID, your X, your Y, and your ID, your X, your Y, and your ID, your X, your Y, and your health. And then your health. And then your health. And then it writes the entity it writes the entity it writes the entity into the transmission channel. into the transmission channel. into the transmission channel. Like, it should just be Like, it should just be Like, it should just be written like that written like that written like that in the title. But in the title. But in the title. But it actually it actually it actually gets complemented, and that's what's gets complemented, and that's what's gets complemented, and that's what's happening here, happening here, happening here, when you read this, when you read this, when you read this, your X and Y values ​​are your X and Y values ​​are your X and Y values ​​are actually being read actually being read actually being read incorrectly. And incorrectly. And incorrectly. And since we can since we can since we can debug the client, debug the client, debug the client, the easiest the easiest the easiest way to do that way to do that way to do that would be to run the would be to run the would be to run the debugger here. And then debugger here. And then debugger here. And then read ours if I read ours if I read ours if I can summon it. Uh can summon it. Uh , let's see , let's see . Let's go one . Let's go one . Let's go one step further. We step further. We step further. We need to go past the need to go past the need to go past the inventory here. Now, inventory here. Now, inventory here. Now, why is it breaking why is it breaking why is it breaking there? Ahem. Now it gets even there? Ahem. Now it gets even there? Ahem. Now it gets even "better". Good. If we "better". Good. If we "better". Good. If we debug, debug, debug, obviously we'll get obviously we'll get obviously we'll get to our right line here and to our right line here and to our right line here and stop, and we'll find stop, and we'll find that these values ​​are that these values ​​are that these values ​​are absurd, because we're absurd, because we're absurd, because we're just outputting them and just outputting them and just outputting them and showing that they're

showing that they're showing that they're just nonsense. just nonsense. The problem is that The problem is that the padding the padding the padding is written over and is written over and is written over and the client reads over the client reads over the client reads over that value because that value because that value because it is padded in it is padded in it is padded in such a way that you such a way that you such a way that you run into this run into this run into this problem. And this bug problem. And this bug problem. And this bug likes to hide in likes to hide in likes to hide in places where it is not places where it is not places where it is not specified in your specified in your specified in your protocol. Your protocol. Your protocol. Your entity in the feed is entity in the feed is typed as an entity record here , where you have an ID, a type, two , where you have an ID, a type, two , where you have an ID, a type, two floating floating floating point numbers, your health, point numbers, your health, point numbers, your health, and then the raw and then the raw and then the raw data. And on the data. And on the data. And on the server side for your server side for your server side for your communication channel, the communication channel, the communication channel, the entity is the ID, type, entity is the ID, type, entity is the ID, type, placeholder X, Y, and placeholder X, Y, and placeholder X, Y, and health. And you can health. And you can health. And you can see what the see what the see what the problem is: problem is: problem is: there is a placeholder on the server side, there is a placeholder on the server side, there is a placeholder on the server side, but there is none on the client side but there is none on the client side but there is none on the client side , and there you , and there you , and there you have found the error. And it's have found the error. And it's have found the error. And it's much harder to detect, much harder to detect, much harder to detect, catch, and fix catch, and fix catch, and fix because it's hard because it's hard because it's hard to prove. It's hard to to prove. It's hard to to prove. It's hard to look at and find the look at and find the look at and find the problem. So this is a problem. So this is a problem. So this is a case where case where case where you run into a you run into a you run into a problem that's problem that's problem that's hard to pinpoint because it's hard to pinpoint because it's hard to pinpoint because it's not about the protocol not about the protocol . This is not a server. This is not . This is not a server. This is not . This is not a server. This is not a client. Both a client. Both a client. Both read and understand read and understand read and understand everything exactly as they should everything exactly as they should . The actual problem is that

. The actual problem is that . The actual problem is that you didn't you didn't you didn't specify in the specify in the specify in the protocol. This is what protocol. This is what protocol. This is what you defined you defined you defined directly in the directly in the directly in the communication channel. So, communication channel. So, communication channel. So, defining more defining more defining more data in the data in the data in the protocol rather than the protocol rather than the protocol rather than the channel solves this channel solves this channel solves this problem. So as long as you problem. So as long as you problem. So as long as you focus on focus on focus on describing everything in the describing everything in the describing everything in the protocol, it becomes a protocol, it becomes a protocol, it becomes a matter of protocol. And matter of protocol. And matter of protocol. And then your then your then your two-way two-way two-way debugging really debugging really debugging really works, and this method is works, and this method is works, and this method is effective. The problem effective. The problem effective. The problem arises when you arises when you arises when you record things that are record things that are record things that are not on the record and not on the record and not on the record and send them. Now, send them. Now, send them. Now, the last thing we'll the last thing we'll the last thing we'll briefly talk about briefly talk about briefly talk about is the crash patterns for is the crash patterns for is the crash patterns for each language. Obviously each language. Obviously , every language has a set of , every language has a set of errors that errors that at least follow a at least follow a at least follow a similar pattern. And similar pattern. And similar pattern. And if you look at the if you look at the if you look at the symptom, you can easily symptom, you can easily symptom, you can easily determine which determine which determine which runtime is to runtime is to runtime is to blame. So, let's blame. So, let's blame. So, let's talk about them. In talk about them. In talk about them. In this situation, I will this situation, I will this situation, I will only talk about C# and only talk about C# and only talk about C# and C++. But by mastering C++. But by mastering C++. But by mastering other languages, you will be able to other languages, you will be able to other languages, you will be able to understand understand understand them more easily on your own. The first them more easily on your own. The first is C#. Tasks are is C#. Tasks are is C#. Tasks are very good at " very good at " swallowing" exceptions.

swallowing" exceptions. swallowing" exceptions. If you never If you never If you never expect expect expect the task to complete, the exception the task to complete, the exception the task to complete, the exception simply disappears. He is simply disappears. He is simply disappears. He is released from released from released from memory. And no memory. And no memory. And no one will hear about him again one will hear about him again . The garbage collector . The garbage collector . The garbage collector likes to pause, likes to pause, likes to pause, and this is often and this is often and this is often perceived as perceived as perceived as network latency network latency network latency because the garbage collector can because the garbage collector can because the garbage collector can slow things down slow things down . This may take time. . This may take time. And when this happens, you And when this happens, you find yourself in a find yourself in a find yourself in a situation where situation where situation where your your your network seems to be slowing down. It seems like network seems to be slowing down. It seems like network seems to be slowing down. It seems like your server is sending your server is sending your server is sending packets slowly. And packets slowly. And packets slowly. And similar situations. You similar situations. You similar situations. You may encounter may encounter may encounter the fact that the execution continuation the fact that the execution continuation occurs in occurs in the wrong thread, and then the wrong thread, and then the wrong thread, and then there is a problem with the there is a problem with the there is a problem with the context. This is context. This is context. This is really hard really hard really hard to identify, but to identify, but to identify, but since the since the since the Rider debugger Rider debugger Rider debugger now allows you now allows you now allows you to work with threads to work with threads to work with threads and variables, you and variables, you and variables, you can very quickly can very quickly can very quickly look at each look at each look at each individual thread that is individual thread that is individual thread that is running and determine running and determine running and determine if something is if something is if something is being restored in being restored in being restored in the wrong thread the wrong thread the wrong thread by checking its by checking its by checking its state and finding the state and finding the state and finding the point at which the point at which the point at which the problem occurred. This is one of the problem occurred. This is one of the problem occurred. This is one of the biggest advantages biggest advantages biggest advantages of using the of using the of using the Rider client and Rider client and Rider client and JetBrains' .NET tools: you JetBrains' .NET tools: you JetBrains' .NET tools: you can very easily can very easily can very easily look at each of look at each of look at each of these individual issues and these individual issues and these individual issues and see if see if see if your your your states diverge. Now, if we states diverge. Now, if we states diverge. Now, if we go back to C++, go back to C++, go back to C++, things are a little more things are a little more things are a little more complicated. C++ and

complicated. C++ and complicated. C++ and native code are very native code are very native code are very difficult to debug. difficult to debug. difficult to debug. You are experiencing the You are experiencing the alignment issues during alignment issues during transmission that we transmission that we transmission that we just showed. You're just showed. You're just showed. You're dealing with dealing with buffer reuse buffer reuse between clocks, which between clocks, which between clocks, which leads to weird leads to weird alignment issues that cause alignment issues that cause the information to be so the information to be so the information to be so distorted distorted . Sometimes aligned . Sometimes aligned . Sometimes aligned information doesn't information doesn't information doesn't cause problems or cause problems or cause problems or crashes crashes crashes because it's because it's because it's filled with zeros, but filled with zeros, but filled with zeros, but if the buffer if the buffer if the buffer is is is reused, you can reused, you can reused, you can write garbled write garbled write garbled data, which also data, which also data, which also creates serious creates serious creates serious problems. And finally, the problems. And finally, the use-after-free error. use-after-free error. If you free If you free If you free an object in a loop and an object in a loop and an object in a loop and another thread another thread another thread continues continues continues to use it, it to use it, it to use it, it looks like looks like looks like data corruption and data corruption and data corruption and is very difficult is very difficult is very difficult to identify. Therefore, to identify. Therefore, to identify. Therefore, I strongly advise you to I strongly advise you to I strongly advise you to spend time spend time spend time tracking the tracking the tracking the usage of malloc and usage of malloc and usage of malloc and free to avoid free to avoid free to avoid such problems. And, such problems. And, such problems. And, actually, that's all. That's the actually, that's all. That's the actually, that's all. That's the whole method. You whole method. You whole method. You set a set a set a breakpoint before breakpoint before breakpoint before you read any you read any you read any code. You correlate a code. You correlate a code. You correlate a shared ID so that shared ID so that shared ID so that both parties have a both parties have a both parties have a piece of information to piece of information to rely on. And then you rely on. And then you read the bytes.

read the bytes. Specifically, you go in Specifically, you go in and read aligned and read aligned and read aligned information or data from the information or data from the information or data from the network when two network when two network when two debuggers debuggers debuggers disagree, disagree, disagree, and check the boundary and check the boundary and check the boundary last, not last, not last, not first. This is exactly what first. This is exactly what first. This is exactly what I want I want I want to emphasize the most to emphasize the most . Checking the protocol . Checking the protocol , network, and boundary , network, and boundary , network, and boundary first forces you first forces you first forces you to spend more to spend more to spend more time debugging the time debugging the time debugging the protocol boundary than protocol boundary than protocol boundary than debugging debugging debugging each side each side each side individually. Take the border individually. Take the border individually. Take the border as the source of truth. as the source of truth. Take it as Take it as an answer to the an answer to the an answer to the problem, and then problem, and then problem, and then move from there to move from there to move from there to each side, each side, each side, depending on which one is depending on which one is depending on which one is telling the truth. telling the truth. If you are reading data If you are reading data from both sides to from both sides to from both sides to find the truth when you find the truth when you find the truth when you set a set a set a breakpoint, then the boundary should breakpoint, then the boundary should breakpoint, then the boundary should be the last thing be the last thing be the last thing you look at because you look at because you look at because you have checked both you have checked both you have checked both sides and they have the sides and they have the answers you need. This method has answers you need. This method has always worked for always worked for always worked for me, and I really me, and I really me, and I really hope it hope it hope it works for you too.

works for you too. After all, checking the boundary After all, checking the boundary last, last, last, making sure it's the making sure it's the making sure it's the last place to last place to last place to look, look, look, placing the code correctly, placing the code correctly, placing the code correctly, matching which matching which matching which package package package corresponds to which, reading corresponds to which, reading corresponds to which, reading bytes when bytes when bytes when debuggers debuggers debuggers contradict each contradict each contradict each other—all of this together other—all of this together other—all of this together is a method that is a method that is a method that helps me a lot. Thank you helps me a lot. Thank you helps me a lot. Thank you all. all. Great report, Great report, Alex. Thank you very much. Alex. Thank you very much. Alex. Thank you very much. Thank you. Thank you. Thank you. Yes, thank you very much, Alex Yes, thank you very much, Alex . Your presentation was very . Your presentation was very . Your presentation was very successful and successful and successful and covered all the covered all the covered all the issues perfectly. We don't issues perfectly. We don't issues perfectly. We don't really have really have really have any questions from the any questions from the any questions from the community. community. community. It was that good. And there It was that good. And there It was that good. And there was a was a was a lot of love for you in the comments lot of love for you in the comments . Your community . Your community . Your community supported you. supported you. supported you. Oh, that's just wonderful. I'm Oh, that's just wonderful. I'm Oh, that's just wonderful. I'm glad they glad they glad they came to came to came to support me. support me. support me. Yes, that was it. Yes, that was it. Yes, that was it. I have a quick I have a quick I have a quick question for you question for you question for you if you have a few if you have a few if you have a few minutes. minutes. minutes. Yes, of course. Yes, of course. I am curious about I am curious about protocol generation protocol generation . Do you prefer the . Do you prefer the source of truth to be source of truth to be on the client or on the client or on the client or server side? Or maybe server side? Or maybe server side? Or maybe you're creating a shared you're creating a shared you're creating a shared library that can be library that can be library that can be used in used in used in both languages ​​to both languages ​​to both languages ​​to create a common create a common create a common abstraction?

abstraction? That's a very good That's a very good question. It's question. It's question. It's interesting to answer, interesting to answer, interesting to answer, because both approaches because both approaches because both approaches look quite look quite look quite attractive. If I attractive. If I attractive. If I place this on place this on place this on the client, then my the client, then my the client, then my client is never client is never client is never wrong, wrong, wrong, right? Or if I right? Or if I right? Or if I host this on the host this on the host this on the server, then when writing server, then when writing server, then when writing functions for the server, functions for the server, functions for the server, it's easier to write the it's easier to write the it's easier to write the protocol part protocol part protocol part there. And a shared there. And a shared there. And a shared library means library means library means that when I write something on the that when I write something on the that when I write something on the client, I client, I client, I have to look have to look have to look into the shared library. into the shared library. into the shared library. If I write something on If I write something on If I write something on the server, I the server, I the server, I have to...well, you have to...well, you have to...well, you get tradeoffs get tradeoffs get tradeoffs on both sides. on both sides. In general, I find that a In general, I find that a shared library shared library is usually the least is usually the least is usually the least favorite way to favorite way to favorite way to solve a problem, solve a problem, solve a problem, since the since the since the problem itself is usually on the problem itself is usually on the problem itself is usually on the client side. And client side. And client side. And if you decide if you decide if you decide to place the protocol to place the protocol to place the protocol on the client, then you on the client, then you on the client, then you have to have to have to adapt adapt adapt the server to it. Your the server to it. Your the server to it. Your server must server must server must comply with this comply with this comply with this protocol. And I protocol. And I protocol. And I like that a little bit like that a little bit like that a little bit more because it's obvious more because it's obvious more because it's obvious that the client is doing that the client is doing that the client is doing the rendering, handling the the rendering, handling the the rendering, handling the reconciliation, and all those reconciliation, and all those reconciliation, and all those little tasks. And little tasks. And little tasks. And if you decide that the if you decide that the if you decide that the client should client should client should provide this, not the provide this, not the provide this, not the server, then you also server, then you also server, then you also get fewer get fewer get fewer errors during errors during errors during execution on the execution on the execution on the client itself. It is much easier for the server client itself. It is much easier for the server to adapt to to adapt to this task. And then, this task. And then, this task. And then, when your server goes down when your server goes down , and not the client, you , and not the client, you , and not the client, you can quickly can quickly can quickly detect it thanks to

detect it thanks to detect it thanks to better better better server-side logging. So server-side logging. So server-side logging. So I would choose the option with I would choose the option with I would choose the option with the client. I would avoid the client. I would avoid the client. I would avoid using a using a using a shared library shared library shared library because it creates unnecessary because it creates unnecessary because it creates unnecessary overhead. overhead. overhead. Of course. Perfectly. Of course. Perfectly. Of course. Perfectly. Thank you for your thoughts Thank you for your thoughts Thank you for your thoughts on this matter. By on this matter. By on this matter. By the way, we got a the way, we got a the way, we got a similar question from the similar question from the similar question from the chat just now: what do you chat just now: what do you chat just now: what do you think about think about think about generated generated generated protocols like Protobuf protocols like Protobuf protocols like Protobuf and similar solutions and similar solutions and similar solutions that take over that take over that take over some of the hard some of the hard some of the hard work? work? I touched on this a bit I touched on this a bit because I wanted to because I wanted to because I wanted to focus on Protobuf, focus on Protobuf, focus on Protobuf, as I know it is as I know it is as I know it is used used used much more often than much more often than much more often than writing a protocol writing a protocol writing a protocol by hand on one by hand on one by hand on one side or the other. I've seen side or the other. I've seen side or the other. I've seen this much more often and this much more often and this much more often and in a much larger in a much larger in a much larger number of projects. And number of projects. And number of projects. And this is actually a good this is actually a good this is actually a good decision. I personally decision. I personally decision. I personally prefer prefer prefer to write the protocol to write the protocol to write the protocol once in a language I once in a language I once in a language I already use. So already use. So , if I'm writing a protocol , if I'm writing a protocol , if I'm writing a protocol for a client, in this for a client, in this for a client, in this case I'll write case I'll write case I'll write it in C#. And then I'll it in C#. And then I'll it in C#. And then I'll generate bindings generate bindings generate bindings for my C++ server for my C++ server for my C++ server so that it reads the so that it reads the so that it reads the same protocol, but same protocol, but same protocol, but it'll be implemented it'll be implemented it'll be implemented exactly the way I want it and in the exactly the way I want it and in the exactly the way I want it and in the language I language I language I need it to be more need it to be more need it to be more natural. And then you natural. And then you natural. And then you will have this will have this will have this broadcast process in just broadcast process in just broadcast process in just one place. Whereas one place. Whereas one place. Whereas with Protobuf—yes, it has with Protobuf—yes, it has with Protobuf—yes, it has bindings for all bindings for all bindings for all possible possible possible programming languages. And programming languages. And programming languages. And for the most part I'm a for the most part I'm a for the most part I'm a fan of that, but fan of that, but fan of that, but not a big fan of Protobuf not a big fan of Protobuf , because that's another place , because that's another place , because that's another place where you can run into a where you can run into a where you can run into a problem. This is another problem. This is another problem. This is another language to deal with

language to deal with . This is another . This is another runtime, well, it's runtime, well, it's runtime, well, it's not even a runtime, but for the not even a runtime, but for the not even a runtime, but for the context of my context of my context of my story, I'll call it that story, I'll call it that story, I'll call it that . This is another . This is another . This is another runtime that runtime that runtime that needs to be needs to be needs to be debugged. And debugged. And debugged. And if you write your if you write your if you write your protocol in C# and then protocol in C# and then protocol in C# and then generate bindings generate bindings generate bindings for C++, then when for C++, then when for C++, then when debugging in C#, you are debugging in C#, you are debugging in C#, you are debugging the debugging the debugging the protocol in C#. protocol in C#. When you run a When you run a protocol in C++, you are protocol in C++, you are protocol in C++, you are debugging a debugging a debugging a protocol in C++. If protocol in C++. If protocol in C++. If both options both options both options fail, you fail, you fail, you have to have to have to resort to resort to resort to Protobuf itself, and this is where some Protobuf itself, and this is where some Protobuf itself, and this is where some discrepancy arises. discrepancy arises. discrepancy arises. So I prefer to So I prefer to So I prefer to work within a single work within a single work within a single language, although I appreciate what language, although I appreciate what language, although I appreciate what Protobuf brings to Protobuf brings to Protobuf brings to development. development. Yes. I think there are also Yes. I think there are also many nuances here: are many nuances here: are many nuances here: are you working on you working on you working on something like a live something like a live service where you have to service where you have to deal with clients and deal with clients and servers of different servers of different servers of different versions, or versions, or versions, or update them update them update them gradually, where not gradually, where not gradually, where not all users all users all users switch to the new switch to the new switch to the new version at the same time; version at the same time; version at the same time; compared to compared to compared to a situation where you a situation where you a situation where you can force people to can force people to can force people to update or update or update or control both control both control both sides of the process.

sides of the process. Certainly. And this is where Certainly. And this is where Protobuf loses out a bit, Protobuf loses out a bit, Protobuf loses out a bit, as you can as you can as you can version version version the protocol well on the client and the protocol well on the client and the protocol well on the client and have the server have the server have the server configured for that configured for that configured for that version. And then it version. And then it version. And then it only connects only connects only connects to the version of the package it to the version of the package it to the version of the package it sends. So if sends. So if sends. So if your client is outdated, your client is outdated, your client is outdated, it still keeps it still keeps it still keeps the version it the version it the version it needs. Although you needs. Although you needs. Although you can build a can build a can build a server to the same server to the same server to the same specification and specification and specification and keep it keep it keep it running, of course, within running, of course, within running, of course, within reasonable time limits. reasonable time limits. Perfectly. Well, thanks for Perfectly. Well, thanks for all your ideas all your ideas all your ideas today, Alex. It was today, Alex. It was today, Alex. It was great to learn and great to learn and great to learn and see the depth of see the depth of see the depth of issues related to issues related to issues related to networks and different networks and different networks and different programming languages. This programming languages. This programming languages. This was a really great was a really great was a really great dive into the topic as dive into the topic as dive into the topic as we wrap up we wrap up we wrap up our day of our day of our day of game development. game development. game development. Class. Thank you very much to everyone who Class. Thank you very much to everyone who Class. Thank you very much to everyone who invited me. invited me. invited me. Yes, thank you, Alex.