Turning My Obsidian Vault Into a Local AI Engineer — Filip Makraduli, Superlinked

Hello, hello. Um, Hello, hello. Um, I think we're starting. I think we're starting. I think we're starting. I know some of you are I know some of you are I know some of you are still joining, but still joining, but still joining, but I'll continue, and you I'll continue, and you I'll continue, and you can join in at can join in at can join in at any time. any time. So, you may So, you may have seen the title of this have seen the title of this have seen the title of this report. What report. What report. What I have on the slides is I have on the slides is I have on the slides is a little different, a little different, a little different, but I promise—it's the but I promise—it's the but I promise—it's the same direction and the same same direction and the same same direction and the same general idea. It general idea. It all started with my all started with my desire to create a desire to create a desire to create a case related to case related to case related to privacy and privacy and privacy and token value. So token value. So , um, I started with Obsidian. , um, I started with Obsidian. , um, I started with Obsidian. I had some I had some I had some markdown files there. Um, and as markdown files there. Um, and as markdown files there. Um, and as you know, there are these you know, there are these you know, there are these wonderful, um, graphs, and it's all wonderful, um, graphs, and it's all wonderful, um, graphs, and it's all very convenient. very convenient. It looks very nice. However, um, I had However, um, I had some problems beyond some problems beyond some problems beyond just just just organizing the organizing the . Many people . Many people . Many people use such use such use such knowledge graphs and so on as knowledge graphs and so on as knowledge graphs and so on as memory for agents, and memory for agents, and memory for agents, and that's pretty good.

that's pretty good. However, I needed to However, I needed to approach this approach this approach this differently because there were differently because there were privacy concerns. When you When you use use use agents, cloud code, agents, cloud code, agents, cloud code, or collaboration, you or collaboration, you or collaboration, you can inadvertently can inadvertently can inadvertently open your files, open your files, open your files, and for some institutions and for some institutions and for some institutions or individuals, this was a or individuals, this was a or individuals, this was a requirement that had to be requirement that had to be requirement that had to be met. So the goal met. So the goal met. So the goal was to keep everything was to keep everything was to keep everything private and on private and on private and on our own server, but our own server, but our own server, but at the same time make it at the same time make it at the same time make it convenient and accessible convenient and accessible convenient and accessible to people. And, um, there was a to people. And, um, there was a to people. And, um, there was a certain openness to certain openness to certain openness to retraining, retraining, retraining, adaptability, adaptability, adaptability, using using using open source models, and I open source models, and I open source models, and I found a solution in found a solution in found a solution in Superlinked—it's an Superlinked—it's an open source model inference engine. It contains both a It contains both a cluster and a cluster and a model inference solution. Today I'll model inference solution. Today I'll talk about that, and talk about that, and talk about that, and how I how I how I managed to combine these managed to combine these managed to combine these concepts and concepts and concepts and create something create something create something based on these based on these based on these three things. The first three things. The first three things. The first main idea was that main idea was that main idea was that I I I wanted to make Claude Code wanted to make Claude Code wanted to make Claude Code a little cheaper and a little cheaper and a little cheaper and save on tokens.

save on tokens. We know how with We know how with the release of Fable there the release of Fable there the release of Fable there was initially a price for the API only, was initially a price for the API only, was initially a price for the API only, which may which may which may have disappointed some, have disappointed some, have disappointed some, or maybe there is now or maybe there is now or maybe there is now more more more focus on tokenomics. That focus on tokenomics. That focus on tokenomics. That was one of the moments. was one of the moments. was one of the moments. Especially when Especially when Especially when processing documents, processing documents, processing documents, sensitive documents, sensitive documents, sensitive documents, where you need to where you need to where you need to use OCR, use OCR, use OCR, scans, images, scans, images, scans, images, etc.—the value of etc.—the value of etc.—the value of tokens can tokens can tokens can increase significantly. So, I wanted to increase significantly. So, I wanted to increase significantly. So, I wanted to find a way to find a way to find a way to do that, but do that, but do that, but having fewer having fewer having fewer artifacts means artifacts means artifacts means better context, which better context, which better context, which limits the limits the limits the loss of context to some extent, and loss of context to some extent, and loss of context to some extent, and you're not filling it with you're not filling it with you're not filling it with data that could be data that could be data that could be processed in some processed in some processed in some other way. Um, and I wanted to other way. Um, and I wanted to other way. Um, and I wanted to make it private. make it private. Yes, for certain Yes, for certain situations it is perfectly situations it is perfectly situations it is perfectly acceptable to put acceptable to put acceptable to put everything into Claude Code or everything into Claude Code or everything into Claude Code or use use use this API there. But what if you this API there. But what if you this API there. But what if you could host everything could host everything could host everything yourself and have yourself and have yourself and have your own VPC or cloud your own VPC or cloud your own VPC or cloud to pre- to pre- to pre- process process process sensitive sensitive sensitive documents and then documents and then documents and then get the result get the result get the result back. So that was back. So that was back. So that was another goal of mine during another goal of mine during another goal of mine during development. I also development. I also development. I also needed needed needed adaptability, so adaptability, so adaptability, so it wasn't just about it wasn't just about it wasn't just about privacy and privacy and privacy and reducing reducing reducing token costs, but also about being token costs, but also about being token costs, but also about being able to test able to test able to test multiple models; multiple models; multiple models; It is possible to deploy It is possible to deploy It is possible to deploy LoRAs that are

LoRAs that are LoRAs that are highly specialized highly specialized , as such , as such , as such sensitive sensitive sensitive tasks usually tasks usually tasks usually require some additional require some additional require some additional training or training or training or adaptation to a adaptation to a adaptation to a specific niche. So, specific niche. So, specific niche. So, I had to I had to I had to do that too. do that too. The setup includes The setup includes Superlinked and the Superlinked and the Superlinked and the psi repository, so for psi repository, so for psi repository, so for today's purpose today's purpose today's purpose you can go with the you can go with the you can go with the repository which is repository which is repository which is completely open source, completely open source, completely open source, so you can find so you can find so you can find it on GitHub. This is Superlinked it on GitHub. This is Superlinked it on GitHub. This is Superlinked psi. Uh, you can, yes. psi. Uh, you can, yes. psi. Uh, you can, yes. So, this is him. This is So, this is him. This is So, this is him. This is a logo. And I'll talk a logo. And I'll talk a logo. And I'll talk about integration, which... about integration, which... Oh, sorry, not here. Oh, sorry, not here. So, the MCP integration, So, the MCP integration, which is essentially a package which is essentially a package which is essentially a package that we created, and this is this one that we created, and this is this one that we created, and this is this one . So you . So you . So you can also look in that can also look in that can also look in that subdirectory. This is what subdirectory. This is what subdirectory. This is what most of most of most of the talk will be about, but the talk will be about, but the talk will be about, but I will also mention why I will also mention why I will also mention why inference inference inference was a bottleneck was a bottleneck was a bottleneck when building when building when building something like this.

something like this. So it was easy So it was easy to understand how to understand how to understand how to store data, to store data, to store data, maybe in Obsidian etc., maybe in Obsidian etc., maybe in Obsidian etc., and have these relationships between the and have these relationships between the and have these relationships between the data. Um, that was the data. Um, that was the data. Um, that was the easy part. I easy part. I easy part. I could also could also could also use the API. use the API. However, if I wanted to However, if I wanted to use use use open-source inference at open-source inference at open-source inference at scale, test scale, test scale, test multiple models, and be multiple models, and be multiple models, and be able to able to able to deploy it in deploy it in deploy it in production, things production, things production, things got a little more got a little more got a little more complicated. And that's why the complicated. And that's why the complicated. And that's why the infrastructural infrastructural infrastructural advantage that advantage that advantage that Sci offers, Sci offers, Sci offers, the ability to the ability to the ability to deploy both deploy both deploy both the cluster and the the cluster and the the cluster and the models themselves, became a big models themselves, became a big models themselves, became a big advantage in my advantage in my advantage in my case. Um, here you case. Um, here you case. Um, here you can see this can see this can see this example from Obsidian. So I'll example from Obsidian. So I'll example from Obsidian. So I'll work with work with work with some of these things. Uh some of these things. Uh , but you , but you , but you can also try it can also try it can also try it yourself. Um, but yourself. Um, but yourself. Um, but I'll I'll I'll use a use a use a managed cluster. We managed cluster. We managed cluster. We can't, um, I can't, um, I can't, um, I can't give it to you right now can't give it to you right now live. However, live. However, you can go to you can go to you can go to our website, this one, our website, this one, our website, this one, and there you can and there you can and there you can apply for apply for data mining grants. So, you can data mining grants. So, you can go there and go there and go there and get your, um, API get your, um, API key to try key to try key to try and do things faster.

and do things faster. and do things faster. If not, you can If not, you can If not, you can work locally work locally work locally since there is a Docker since there is a Docker container and all that. container and all that. container and all that. Well, it's going to take a little Well, it's going to take a little Well, it's going to take a little longer, but I'd longer, but I'd longer, but I'd rather you rather you rather you listen to me now and listen to me now and listen to me now and then try it then try it then try it yourself, although you can do it yourself, although you can do it yourself, although you can do it in parallel if in parallel if in parallel if you want. And we also have these you want. And we also have these you want. And we also have these really cool, um, kind of really cool, um, kind of really cool, um, kind of memorable QR codes that you memorable QR codes that you memorable QR codes that you can use to can use to can use to get an even better get an even better get an even better cluster, in my opinion, and cluster, in my opinion, and cluster, in my opinion, and you can find them you can find them you can find them here, uh, after the talk here, uh, after the talk . So, um, as I . So, um, as I . So, um, as I said, the idea was to said, the idea was to provide an element of provide an element of privacy and a privacy and a privacy and a way to offload way to offload way to offload tasks. So, actually, tasks. So, actually, tasks. So, actually, I didn't want to disrupt I didn't want to disrupt I didn't want to disrupt my usual my usual my usual daily daily daily workflow. Um, so I workflow. Um, so I workflow. Um, so I just work in the just work in the just work in the cloud code, and if cloud code, and if cloud code, and if there's a specific there's a specific there's a specific task that task that task that needs to be done in needs to be done in needs to be done in my virtual my virtual my virtual private clouds that private clouds that private clouds that needs to be processed needs to be processed needs to be processed that's sensitive that's sensitive that's sensitive and worth a lot of and worth a lot of and worth a lot of tokens, I can tokens, I can tokens, I can just offload just offload just offload it and keep it and keep it and keep working. So the natural working. So the natural working. So the natural solution was to solution was to solution was to create some kind of, um, create some kind of, um, create some kind of, um, skill or plugin. But skill or plugin. But skill or plugin. But this was further this was further this was further enhanced by enhanced by enhanced by creating an entire creating an entire creating an entire MCP server that MCP server that MCP server that is called in the is called in the is called in the cloud code, and there is an cloud code, and there is an cloud code, and there is an element of element of element of skill usage here, as well as skill usage here, as well as skill usage here, as well as having a having a having a specialized specialized specialized node that node that node that can really do the can really do the can really do the heavy lifting. And you heavy lifting. And you heavy lifting. And you can also can also can also use use

Superlinks data output, which I'll Superlinks data output, which I'll talk about later, talk about later, talk about later, including how we including how we including how we did it and what did it and what did it and what the benefits are of the benefits are of the benefits are of using the using the using the Superlinks output engine. This Superlinks output engine. This Superlinks output engine. This saves tokens saves tokens saves tokens because, for example, because, for example, because, for example, you can take 10 you can take 10 you can take 10 pages of a PDF file pages of a PDF file pages of a PDF file containing images containing images containing images and various details. So and various details. So and various details. So if you if you if you preprocess this preprocess this preprocess this with good quality with good quality open open open source OCR models and then source OCR models and then source OCR models and then go back to your go back to your go back to your main agent or main agent or main agent or cloud code or cloud code or cloud code or whatever you're whatever you're whatever you're using, you using, you using, you get the context and get the context and get the context and also the benefit of also the benefit of also the benefit of lower lower lower token costs, better token costs, better token costs, better context, etc. So, context, etc. So, context, etc. So, that's the general idea, that's the general idea, that's the general idea, especially for OCR it's especially for OCR it's especially for OCR it's obvious. However, obvious. However, obvious. However, there are numerous there are numerous benefits to saving benefits to saving tokens even beyond tokens even beyond tokens even beyond OCR. So why OCR. So why OCR. So why MCP and not just an API MCP and not just an API MCP and not just an API or plugin? Since or plugin? Since or plugin? Since we have a dedicated we have a dedicated we have a dedicated cluster, it was cluster, it was cluster, it was much better much better much better to divide the responsibilities and to divide the responsibilities and to divide the responsibilities and have this as a basis for have this as a basis for have this as a basis for further development of further development of further development of additional details additional details additional details and features, which I will and features, which I will and features, which I will also talk about also talk about also talk about later. And everything runs later. And everything runs later. And everything runs on a server that you on a server that you on a server that you can control, can control, can control, so through Sai you so through Sai you so through Sai you can choose can choose can choose worker pools, choose worker pools, choose worker pools, choose your GPUs, and understand your GPUs, and understand your GPUs, and understand which models which models which models are assigned where. This

are assigned where. This are assigned where. This flexibility from a flexibility from a flexibility from a cloud perspective was cloud perspective was cloud perspective was an advantage, and I wanted an advantage, and I wanted an advantage, and I wanted to keep that, so I to keep that, so I to keep that, so I chose MCP over just chose MCP over just chose MCP over just Skill. Um, and it Skill. Um, and it Skill. Um, and it opens up access to the opens up access to the opens up access to the appropriate appropriate appropriate tools. So, this is tools. So, this is tools. So, this is roughly how it would roughly how it would roughly how it would work if it work if it work if it were just a plugin on were just a plugin on were just a plugin on your laptop or your laptop or your laptop or device. So device. So you can take you can take you can take that experience to the cloud, that experience to the cloud, that experience to the cloud, which is your own, and which is your own, and which is your own, and control everything there. control everything there. So it's a much more So it's a much more convenient convenient convenient setup. Um, setup. Um, setup. Um, also a lot of also a lot of also a lot of the examples here the examples here the examples here are about PII, which is are about PII, which is are about PII, which is personal information, and personal information, and personal information, and there is a tool to there is a tool to there is a tool to edit edit edit specific details specific details specific details in documents, which is in documents, which is in documents, which is one of the key one of the key one of the key aspects of privacy aspects of privacy . Um, and the MCP setting . Um, and the MCP setting . Um, and the MCP setting has three levels. So, has three levels. So, has three levels. So, in the middle you see in the middle you see in the middle you see Sci MCP. There is an MCP client that Sci MCP. There is an MCP client that Sci MCP. There is an MCP client that acts as a client and acts as a client and acts as a client and host through co-work, cloud code host through co-work, cloud code host through co-work, cloud code or whatever, and it or whatever, and it or whatever, and it interacts with the interacts with the interacts with the Sci MCP server. And then Sci MCP server. And then Sci MCP server. And then you have a Sci cluster with GPUs you have a Sci cluster with GPUs you have a Sci cluster with GPUs and worker pools and worker pools and worker pools that do the work, that do the work, that do the work, interacting with the interacting with the interacting with the MCP server, and you have MCP server, and you have MCP server, and you have different different different models available, and a models available, and a models available, and a queuing mechanism for those queuing mechanism for those queuing mechanism for those GPUs, which I'll talk

GPUs, which I'll talk GPUs, which I'll talk more about later. more about later. The idea of ​​MCP, as you know, is to have a single have a single interface for interface for interface for communication, communication, communication, especially as we especially as we especially as we build build build more and more more and more model routing tools, because the demo and model routing tools, because the demo and current version current version current version are focused on are focused on are focused on calling tools calling tools . However, we are working . However, we are working . However, we are working on routing on routing on routing models and selecting models and selecting models and selecting appropriate models appropriate models appropriate models for different tasks for different tasks for different tasks based on estimates and based on estimates and based on estimates and cost. And all of these cost. And all of these cost. And all of these models will be models will be models will be open and open and open and will run on will run on will run on your GPUs. So, um, that's your GPUs. So, um, that's your GPUs. So, um, that's also an important also an important also an important direction. Hmm, so that's a direction. Hmm, so that's a direction. Hmm, so that's a forward-looking approach forward-looking approach . Um, and here there is a host, a . Um, and here there is a host, a . Um, and here there is a host, a client, and a server. Um, client, and a server. Um, client, and a server. Um, you can do this you can do this you can do this with any with any with any setup you setup you setup you want, whether it's want, whether it's want, whether it's cloud desktop, cloud desktop, cloud desktop, cloud code, etc. And cloud code, etc. And cloud code, etc. And the communication the communication the communication is done is done is done using JSON RPC. So using JSON RPC. So using JSON RPC. So there are three types: response, there are three types: response, there are three types: response, request, and notification, request, and notification, request, and notification, just to just to just to initialize and initialize and initialize and perform the " perform the " handshake" protocol so handshake" protocol so handshake" protocol so you can you can you can authenticate the MCP edge, authenticate the MCP edge, authenticate the MCP edge, etc. And, um, the important etc. And, um, the important etc. And, um, the important detail is that detail is that the server opens up the server opens up the server opens up access to different access to different access to different things. So, the things. So, the things. So, the MCP standard has changed somewhat since

MCP standard has changed somewhat since MCP standard has changed somewhat since its its its inception. But inception. But inception. But the main thing is that it the main thing is that it the main thing is that it provides tools provides tools provides tools that cloud code can then that cloud code can then that cloud code can then use use use . However, . However, . However, in the MCP universe in general, in the MCP universe in general, in the MCP universe in general, there are situations there are situations there are situations where the client where the client where the client can also open can also open can also open things like fetches or things like fetches or things like fetches or queries, making this a queries, making this a queries, making this a two-way two-way two-way communication rather than a communication rather than a communication rather than a standard standard client-server scheme. However, client-server scheme. However, client-server scheme. However, in our case, in our case, in our case, since this was a since this was a privacy setting, we do privacy setting, we do n't do this. So, n't do this. So, n't do this. So, essentially, the MCP edge side essentially, the MCP edge side essentially, the MCP edge side just provides just provides just provides tools that tools that tools that can be can be can be used used used if they were if they were if they were initialized initialized initialized during the handshake, and during the handshake, and during the handshake, and that's it. Therefore, it that's it. Therefore, it that's it. Therefore, it only performs such only performs such only performs such responses, since in responses, since in responses, since in our case it would not our case it would not our case it would not perform more perform more perform more complex tasks. And complex tasks. And complex tasks. And the idea is the idea is the idea is that the tools that the tools that the tools are revealed and you are revealed and you are revealed and you just keep working with just keep working with just keep working with them. Basically, them. Basically, them. Basically, you have tools you have tools you have tools that give you different that give you different that give you different capabilities, and you capabilities, and you capabilities, and you can run them.

can run them. Another detail Another detail is that this is that this is that this can be done can be done can be done locally, so in the locally, so in the locally, so in the MCP universe, MCP universe, MCP universe, standard input- standard input- output (IO) is intended output (IO) is intended output (IO) is intended for local for local for local development. However, we development. However, we development. However, we wanted to wanted to wanted to use use use streaming HTTP streaming HTTP responses. This used to responses. This used to responses. This used to be done through two be done through two be done through two endpoints: one endpoints: one endpoints: one for for for streaming and one for the streaming and one for the streaming and one for the HTTP handshake, but HTTP handshake, but HTTP handshake, but now there is a standard now there is a standard now there is a standard that does this that does this that does this simultaneously, which makes the simultaneously, which makes the simultaneously, which makes the job more convenient. Here are the job more convenient. Here are the job more convenient. Here are the tools we tools we tools we have, so I'll spend have, so I'll spend have, so I'll spend a little more time on this a little more time on this a little more time on this slide. Currently, slide. Currently, slide. Currently, the settings and things the settings and things the settings and things you can you can you can try are related try are related try are related to these models. So, to these models. So, to these models. So, there is a tool for there is a tool for there is a tool for converting converting converting documents to markdown, and documents to markdown, and documents to markdown, and it uses it uses it uses OCR models. These are the open OCR models. These are the open OCR models. These are the open models we have models we have models we have at the moment. The idea at the moment. The idea at the moment. The idea is that this is that this is that this saves on saves on saves on token costs and also token costs and also token costs and also uses a very high- uses a very high- uses a very high- quality quality quality open source model.

open source model. The detail here is that The detail here is that with OCR models, with OCR models, with OCR models, especially especially especially open ones, you do not open ones, you do not open ones, you do not lose quality lose quality lose quality by delegating some of the by delegating some of the by delegating some of the work to such OCR work to such OCR models. So it's not like models. So it's not like I'm using I'm using I'm using open source and open source and open source and sacrificing something. This is not the sacrificing something. This is not the sacrificing something. This is not the case at all, especially here, and case at all, especially here, and case at all, especially here, and in general, in general, in general, open source models open source models open source models are catching up with the leaders are catching up with the leaders , so finding a way , so finding a way , so finding a way to implement them in to implement them in to implement them in production is production is production is quite useful and quite useful and quite useful and important. There is an important. There is an important. There is an entity extraction feature entity extraction feature entity extraction feature that simply generates that simply generates that simply generates relevant entities relevant entities relevant entities in documents in documents in documents using various using various using various NER models. There are also NER models. There are also NER models. There are also various tools various tools various tools for creating for creating for creating captions and captions and captions and tagging. There is tagging. There is tagging. There is also functionality for also functionality for also functionality for generating structure, generating structure, generating structure, summarizing, and summarizing, and summarizing, and answering answering answering questions. And the good thing is questions. And the good thing is questions. And the good thing is that it's open source, that it's open source, that it's open source, so you can so you can so you can do a PR and add do a PR and add do a PR and add another tool another tool another tool right now and we right now and we right now and we can make it can make it can make it part of this and it will part of this and it will part of this and it will still still still use use use inference, which is a inference, which is a inference, which is a bigger bottleneck bigger bottleneck bigger bottleneck in in in situations like this. And that's the situations like this. And that's the situations like this. And that's the idea. You idea. You idea. You can clone the can clone the can clone the repository I repository I repository I showed earlier and showed earlier and showed earlier and run the cluster run the cluster run the cluster locally, but also locally, but also locally, but also if you go back if you go back if you go back to the site and make a

to the site and make a to the site and make a request, you will eventually request, you will eventually request, you will eventually get an API key. This get an API key. This get an API key. This will be a kind of will be a kind of will be a kind of setup. You setup. You setup. You need to get an API need to get an API key. You can also key. You can also key. You can also host this on host this on host this on your own server your own server your own server if you plan on if you plan on if you plan on running it running it running it long term. So it long term. So it long term. So it doesn't have to doesn't have to doesn't have to be our cluster, it be our cluster, it be our cluster, it could be your cluster could be your cluster . And you just . And you just . And you just need to need to need to set it up. I will also set it up. I will also set it up. I will also publish an article that publish an article that publish an article that explains this in more explains this in more explains this in more detail and you detail and you detail and you can get can get can get everything up and running step by step. everything up and running step by step. So take your time. You So take your time. You can can can enjoy enjoy enjoy the performance, the performance, the performance, relax and relax and relax and try everything at your try everything at your try everything at your leisure. Here we leisure. Here we leisure. Here we have a repository, as well have a repository, as well have a repository, as well as links to as links to as links to obtain obtain obtain computing computing computing power and a power and a power and a cluster. You can cluster. You can cluster. You can submit a request, submit a request, submit a request, get a form, get a form, get a form, fill it out, and then they fill it out, and then they fill it out, and then they will send you an API key. will send you an API key. This is exactly the This is exactly the setup I setup I setup I will be going through and will be going through and will be going through and testing.

testing. I hope the demo gods I hope the demo gods are kind. are kind. are kind. Someone mentioned this as Someone mentioned this as Someone mentioned this as something important during something important during something important during one of the presentations. one of the presentations. one of the presentations. I like this I like this I like this idea. And yes, this is just a idea. And yes, this is just a idea. And yes, this is just a setup that I have setup that I have setup that I have already done to some extent already done to some extent already done to some extent . And now I . And now I . And now I will move on to the will move on to the will move on to the demonstration. So, I'm demonstration. So, I'm demonstration. So, I'm moving on to moving on to moving on to cloud code. I cloud code. I cloud code. I am in the am in the am in the appropriate appropriate appropriate package directory. package directory. So I can say, " Okay, can you..." Okay, can you..." Okay, can you..." I have a set of files I have a set of files I have a set of files that are in my that are in my that are in my inbox. I'll inbox. I'll inbox. I'll show you now. Here, show you now. Here, show you now. Here, for example, is a graph, for example, is a graph, for example, is a graph, here is an inbox, here is an inbox, here is an inbox, imagine that these are different imagine that these are different imagine that these are different emails. This is emails. This is emails. This is all synthetic data, all synthetic data, all synthetic data, but relevant. I'm not but relevant. I'm not but relevant. I'm not using the actual using the actual using the actual data that inspired data that inspired data that inspired this project, but this project, but this project, but of course it's things like, " of course it's things like, " Okay, there are some Okay, there are some Okay, there are some invoice emails invoice emails ." There's ." There's ." There's confidential confidential confidential information everywhere here, and you information everywhere here, and you information everywhere here, and you can just can just can just walk through it all walk through it all walk through it all . There are some . There are some . There are some non-disclosure agreements non-disclosure agreements non-disclosure agreements where there is, for example, a where there is, for example, a where there is, for example, a phone number or a phone number or a phone number or a social social social security number. This is what security number. This is what security number. This is what this this this data looks like. And, for example, data looks like. And, for example, data looks like. And, for example, you can come in here and

you can come in here and you can come in here and say, "Can you say, "Can you say, "Can you cross out the data from cross out the data from cross out the data from this this this non-disclosure document?" Something non-disclosure document?" Something non-disclosure document?" Something like this. like this. Cool, we get Cool, we get three credits. So, what three credits. So, what three credits. So, what it looks like, one it looks like, one it looks like, one example—here's this one example—here's this one . This is something I've . This is something I've . This is something I've tried before. This is tried before. This is tried before. This is a deal, and you get a a deal, and you get a a deal, and you get a kind of kind of kind of crossed-out document crossed-out document , where the , where the , where the relevant details are removed, relevant details are removed, relevant details are removed, but, of course, but, of course, but, of course, only the email only the email only the email and the person remain, and other and the person remain, and other and the person remain, and other entities are hidden. And entities are hidden. And entities are hidden. And all of this is done in all of this is done in all of this is done in your VPC. So Cloud Code your VPC. So Cloud Code your VPC. So Cloud Code doesn't see these things, nor does doesn't see these things, nor does doesn't see these things, nor does any agent any agent any agent you you you use. In use. In use. In certain situations, you certain situations, you certain situations, you even get even get even get better performance better performance better performance with OCR. Ahem. I. For example, with OCR. Ahem. I. For example, with OCR. Ahem. I. For example, here you see a successfully here you see a successfully here you see a successfully loaded skill. I loaded skill. I loaded skill. I found a found a found a non-disclosure document. Here I non-disclosure document. Here I non-disclosure document. Here I configured the cluster by configured the cluster by configured the cluster by simply entering the simply entering the simply entering the endpoint and API endpoint and API key. You'll need to key. You'll need to key. You'll need to do it a little do it a little do it a little differently, but you differently, but you differently, but you can still query can still query can still query even in Cloud Code, etc. And even in Cloud Code, etc. And even in Cloud Code, etc. And here we get: here we get: here we get: okay, the file contains okay, the file contains okay, the file contains the first crossed out the first crossed out the first crossed out output, 10 output, 10 output, 10 ranges crossed out from this ranges crossed out from this ranges crossed out from this to this. And this is a kind of

to this. And this is a kind of tag counting. This gives us tag counting. This gives us some information. some information. Maybe we can Maybe we can do something do something do something like, "Can you like, "Can you like, "Can you preprocess preprocess preprocess the scans and also the scans and also the scans and also cross them out?"—that's cross them out?"—that's cross them out?"—that's natural language. We're natural language. We're natural language. We're trying to get the trying to get the trying to get the OCR models working, and OCR models working, and OCR models working, and it's a tool to it's a tool to it's a tool to call, so you call, so you call, so you can also set up can also set up can also set up tool calls tool calls tool calls and all that. There and all that. There and all that. There is also a status bar here that is also a status bar here that is also a status bar here that counts saved counts saved counts saved tokens, but I tokens, but I tokens, but I need to need to need to update it a bit to make it update it a bit to make it update it a bit to make it work correctly; this is work correctly; this is work correctly; this is also part of also part of also part of the advantage as you the advantage as you the advantage as you see savings on the see savings on the see savings on the cost of tokens. So cost of tokens. So cost of tokens. So I preprocess I preprocess I preprocess it in markdown, then it in markdown, then it in markdown, then strikethrough, and it works; strikethrough, and it works; strikethrough, and it works; for example, I'll just for example, I'll just for example, I'll just show you a few show you a few show you a few files so you don't have to files so you don't have to files so you don't have to go through everything all the time go through everything all the time go through everything all the time . There's this, . There's this, . There's this, um, let's say, um, let's say, um, let's say, "before" contract, this is what it "before" contract, this is what it "before" contract, this is what it looks like, this is a scan. I don't looks like, this is a scan. I don't looks like, this is a scan. I don't know if you can see it know if you can see it know if you can see it now because it's clean, now because it's clean, now because it's clean, but it's an image, but it's an image, but it's an image, not text, so it not text, so it not text, so it probably needs OCR.

probably needs OCR. And then if we And then if we do an "after do an "after " version, it's kind of like what you " version, it's kind of like what you " version, it's kind of like what you see here is a see here is a see here is a markdown file. um, it extracts markdown file. um, it extracts markdown file. um, it extracts data, but it also data, but it also data, but it also hides it, oh, hides it, oh, hides it, oh, cursor. So, the idea is to cursor. So, the idea is to cursor. So, the idea is to pre- pre- pre- process the document using an optical process the document using an optical character recognition character recognition (OCR) tool. You get (OCR) tool. You get (OCR) tool. You get token savings, token savings, token savings, better better better context management, etc., but context management, etc., but context management, etc., but also a also a also a redacted file redacted file redacted file so that personal so that personal so that personal information doesn't information doesn't information doesn't leak. And this leak. And this leak. And this is also done is also done is also done in parallel. This is an idea from a in parallel. This is an idea from a in parallel. This is an idea from a functional point of functional point of functional point of view, but there is view, but there is a lot more going on in the background. That is, it's a lot more going on in the background. That is, it's not just about the not just about the privacy aspect, which privacy aspect, which looks easy, looks easy, this this part does the main work. Superlinked, the part does the main work. Superlinked, the part does the main work. Superlinked, the inference engine, and the cluster inference engine, and the cluster inference engine, and the cluster are configured to are configured to are configured to allow for allow for allow for multiple models, multiple models, hot swapping of hot swapping of hot swapping of models across GPUs, models across GPUs, models across GPUs, and smarter batch and smarter batch and smarter batch processing for processing for processing for testing multiple testing multiple testing multiple models simultaneously.

models simultaneously. For example, if you For example, if you have five LoRAs have five LoRAs have five LoRAs configured for configured for configured for different subtasks of different subtasks of different subtasks of your work, you your work, you your work, you can switch can switch can switch between them and test between them and test which one is best. which one is best. which one is best. We have a gateway that We have a gateway that We have a gateway that helps us helps us helps us connect to connect to connect to different worker pools different worker pools different worker pools , each of , each of , each of which has a batch which has a batch which has a batch processor that allows us processor that allows us processor that allows us to perform to perform to perform batching for batching for batching for each worker each worker each worker node separately, which gave node separately, which gave node separately, which gave us better us better us better performance. And this, performance. And this, performance. And this, of course, is the architecture of the of course, is the architecture of the of course, is the architecture of the MCP solution. How it MCP solution. How it MCP solution. How it works with Sai. You can works with Sai. You can works with Sai. You can see that this part see that this part see that this part on the right is on the right is on the right is in your VPC. So, the in your VPC. So, the in your VPC. So, the MCP work that is MCP work that is MCP work that is being ported to being ported to being ported to Superlinked Sai is being done Superlinked Sai is being done Superlinked Sai is being done in your private in your private in your private cloud, and that is what gave cloud, and that is what gave cloud, and that is what gave us the us the us the performance advantage, performance advantage, performance advantage, at least through the at least through the at least through the batching technique batching technique batching technique without using without using without using one common one common one common queue for all nodes queue for all nodes . During testing,

. During testing, . During testing, performance was performance was performance was lower, so we lower, so we lower, so we were able to implement were able to implement were able to implement this setting and this setting and this setting and improve latency improve latency improve latency and performance and performance and performance even across different even across different even across different models. The idea is also that models. The idea is also that models. The idea is also that you need you need you need different node pools because, different node pools because, different node pools because, for example, you might for example, you might for example, you might want to try a want to try a want to try a heavier model, but heavier model, but heavier model, but have another one that have another one that have another one that works great on an L4 instance works great on an L4 instance works great on an L4 instance so you don't waste so you don't waste so you don't waste resources on more expensive resources on more expensive resources on more expensive hardware. hardware. hardware. We also have autoscaling, We also have autoscaling, We also have autoscaling, so the system can so the system can so the system can reduce reduce reduce capacity when it is capacity when it is capacity when it is not being used. not being used. So, this is a setting , and it supports , and it supports hot swapping of hot swapping of hot swapping of models. So, models. So, models. So, especially with smaller especially with smaller especially with smaller models that models that models that fit on a single fit on a single fit on a single GPU GPU GPU and do things and do things and do things like like named entity recognition , or even small , or even small , or even small generation or generation or generation or embeddings, etc., you embeddings, etc., you embeddings, etc., you can gain can gain can gain significant benefit significant benefit significant benefit by switching and by switching and by switching and replacing multiple replacing multiple replacing multiple models on a single GPU.

models on a single GPU. This is in line with This is in line with what what what MCP does and the tooling aspect MCP does and the tooling aspect , where you can create , where you can create agents and carry over agents and carry over agents and carry over some of the inference some of the inference some of the inference using super linked, using super linked, using super linked, setting up different setting up different setting up different task packages. That is task packages. That is , essentially, parsing , essentially, parsing , essentially, parsing documents, as I documents, as I documents, as I explained, or explained, or explained, or creating creating creating content embeddings content embeddings , or writing code , or writing code , or writing code using some using some using some generative models generative models , or extracting , or extracting , or extracting images or images or images or signatures from data, and so on signatures from data, and so on signatures from data, and so on . Or even . Or even guardrails and structural guardrails and structural generation for generation for generation for policy verification, policy verification, policy verification, etc. I would also be etc. I would also be etc. I would also be happy if you happy if you happy if you received more received more received more credits, credits, credits, for example, if you built for example, if you built for example, if you built a project, etc. a project, etc. So, if you contribute a tool, perhaps tool, perhaps for content protection, for content protection, for content protection, policy verification, policy verification, policy verification, or something else that or something else that or something else that you find useful, you find useful, you find useful, we'd be happy we'd be happy we'd be happy to accept it as a contribution to accept it as a contribution . In some of our . In some of our . In some of our tests, we also tests, we also tests, we also saw that thanks to saw that thanks to saw that thanks to this infrastructure this infrastructure this infrastructure that I talked about, we that I talked about, we that I talked about, we get very get very get very good good good performance of performance of performance of embedding models.

embedding models. We also have We also have generative models, generative models, generative models, but the tests and but the tests and but the tests and benchmarks we benchmarks we benchmarks we have currently are have currently are have currently are specifically for specifically for specifically for embedding models. embedding models. You get You get about 10 times about 10 times about 10 times the price at very the price at very the price at very similar quality. You similar quality. You similar quality. You can see that can see that can see that the best model the best model the best model we had for the we had for the we had for the embedding tasks achieved embedding tasks achieved embedding tasks achieved 99.8% of the state- 99.8% of the state- 99.8% of the state- of-the-art models. And of-the-art models. And of-the-art models. And we assessed this we assessed this we assessed this using MTEB tasks. using MTEB tasks. So, yes, that's exactly it. So, yes, that's exactly it. We will also We will also We will also be posting more be posting more be posting more benchmarks and details benchmarks and details benchmarks and details on this soon. But the point is, on this soon. But the point is, on this soon. But the point is, sometimes, sometimes, sometimes, especially in especially in especially in production scenarios production scenarios production scenarios and under certain and under certain and under certain workloads, you workloads, you workloads, you gain an advantage gain an advantage gain an advantage through end-to-end through end-to-end through end-to-end improvement. If your improvement. If your improvement. If your infrastructure and infrastructure and infrastructure and architecture architecture architecture are well-tuned, you are well-tuned, you are well-tuned, you get get get latency improvements and better latency improvements and better latency improvements and better performance, performance, performance, even if the inference of a even if the inference of a even if the inference of a single model is not single model is not single model is not optimized. In optimized. In optimized. In our case, we our case, we our case, we do both, do both, do both, but this was but this was but this was a discovery that a discovery that a discovery that turned out to be very turned out to be very turned out to be very useful for useful for useful for production scenarios production scenarios . Using the example of LoRA, we . Using the example of LoRA, we . Using the example of LoRA, we saw that you can saw that you can create and create and train them quite quickly. So we

train them quite quickly. So we train them quite quickly. So we are exploring work in are exploring work in are exploring work in this direction. Some this direction. Some this direction. Some colleagues were working on colleagues were working on German-specific LoRAs that German-specific LoRAs that were pre-trained, and it were pre-trained, and it were pre-trained, and it turned out to be quite turned out to be quite turned out to be quite cheap. So having a cheap. So having a cheap. So having a way to way to way to use use use multiple LoRAs, test multiple LoRAs, test multiple LoRAs, test them in production, and them in production, and them in production, and deploy them quickly deploy them quickly is very important. is very important. seemed quite seemed quite useful and improved useful and improved useful and improved performance in performance in performance in very narrow niche very narrow niche very narrow niche cases, like cases, like cases, like German legal German legal German legal language, say. And that's pretty much how language, say. And that's pretty much how MCP works. I mentioned MCP works. I mentioned this a little bit. And there are some this a little bit. And there are some this a little bit. And there are some failure modes that already failure modes that already failure modes that already exist, but again exist, but again exist, but again , it works and it works , it works and it works , it works and it works well, but there's still a well, but there's still a well, but there's still a lot that needs to be lot that needs to be lot that needs to be done. And I would be very done. And I would be very done. And I would be very happy to receive your happy to receive your happy to receive your feedback, PRs, and even feedback, PRs, and even feedback, PRs, and even just comments. just comments.

So, you can So, you can add me on LinkedIn and add me on LinkedIn and add me on LinkedIn and stuff, and I'm always happy to stuff, and I'm always happy to stuff, and I'm always happy to chat. And that's the chat. And that's the chat. And that's the idea that the idea that the idea that the same type of MCP edge same type of MCP edge same type of MCP edge handles any handles any document task. So you document task. So you can answer can answer can answer questions, questions, questions, extract facts, extract facts, extract facts, read documents. Um, so read documents. Um, so read documents. Um, so yes, that yes, that yes, that should work too, and should work too, and should work too, and again, again, again, various various various labels and such are removed. And you labels and such are removed. And you labels and such are removed. And you can try it like this can try it like this can try it like this , I think the , I think the , I think the number has changed, but number has changed, but number has changed, but again, you again, you again, you can test can test can test it in real time it in real time it in real time and test different and test different and test different tools with tools with tools with your own demo your own demo data. And I explained it data. And I explained it data. And I explained it to some extent and what to some extent and what to some extent and what it does now. it does now. Um, and yes, there is the Um, and yes, there is the token saving aspect, and token saving aspect, and token saving aspect, and that is very important, and that is very important, and that is very important, and also not sending also not sending also not sending data to external APIs, data to external APIs, data to external APIs, and it all runs in and it all runs in and it all runs in your cloud. Um, and there are your cloud. Um, and there are your cloud. Um, and there are many things to many things to many things to do in the do in the do in the future. Um, but the future. Um, but the future. Um, but the operational operational operational capability is still there capability is still there , so we have Helm charts , so we have Helm charts , so we have Helm charts and a Kubernetes cluster, and you and a Kubernetes cluster, and you and a Kubernetes cluster, and you can also can also can also use use use Terraform apply to Terraform apply to Terraform apply to deploy the cluster deploy the cluster , which is also , which is also , which is also open source, so open source, so open source, so it's not hidden away like it's not hidden away like it's not hidden away like just open source, just open source, just open source, and we have basically and we have basically and we have basically everything. Um, and it can everything. Um, and it can everything. Um, and it can go beyond just

go beyond just go beyond just documents. So documents. So documents. So if we look at this if we look at this if we look at this idea of ​​solving idea of ​​solving idea of ​​solving output as a output as a output as a little box in little box in little box in your your your workflow, it opens up workflow, it opens up workflow, it opens up a lot of possibilities a lot of possibilities a lot of possibilities that can go that can go that can go beyond text with other beyond text with other beyond text with other models models models looking at looking at looking at transcripts, even transcripts, even transcripts, even graph data, graph data, graph data, generating ontologies, generating ontologies, generating ontologies, etc. through etc. through etc. through different different different tool calls, tool calls, tool calls, structuring structuring structuring generation. So there are generation. So there are generation. So there are different possibilities, and different possibilities, and different possibilities, and one of the directions one of the directions one of the directions that we're also that we're also that we're also exploring is exploring is exploring is routing routing routing models in your models in your models in your own private own private own private cloud, which is quite cloud, which is quite cloud, which is quite powerful, and I think the powerful, and I think the powerful, and I think the industry is industry is industry is moving in that moving in that moving in that direction a little bit, where direction a little bit, where direction a little bit, where teams are typically teams are typically teams are typically exploring this exploring this API-based optimization, where you have a API-based optimization, where you have a combination of models combination of models combination of models working together, and the working together, and the working together, and the end-to-end end-to-end end-to-end performance is performance is performance is better. Hmm, and there is an better. Hmm, and there is an better. Hmm, and there is an option to do something option to do something option to do something similar this similar this similar this way. However, this is way. However, this is way. However, this is more of an idea for a more of an idea for a more of an idea for a future future future roadmap, so it's roadmap, so it's roadmap, so it's not built in yet, but not built in yet, but not built in yet, but the idea is to the idea is to the idea is to also select also select also select models depending on the models depending on the models depending on the task. So, the task. So, the task. So, the tool call tool call tool call will be tied not will be tied not will be tied not only to the only to the only to the tool itself, but tool itself, but tool itself, but will also allow you to choose will also allow you to choose will also allow you to choose the model that is best the model that is best the model that is best suited to the suited to the suited to the task.

task. task. That is, if I have That is, if I have That is, if I have only one only one only one OCR model configured, I can OCR model configured, I can OCR model configured, I can say, "Okay, for say, "Okay, for say, "Okay, for this this this tool call, I want tool call, I want tool call, I want to use the cheaper to use the cheaper to use the cheaper OCR." Or any OCR." Or any OCR." Or any other other other tool challenge that tool challenge that tool challenge that adapts to my adapts to my adapts to my needs depending on the needs depending on the task at hand. And this is a very task at hand. And this is a very promising promising promising direction. You can also direction. You can also direction. You can also experiment with experiment with experiment with intents and a intents and a intents and a mix of agent approaches, as mix of agent approaches, as mix of agent approaches, as another strength of another strength of another strength of Sci is Sci is Sci is that if the models are that if the models are that if the models are already loaded into already loaded into already loaded into your GPU, you can your GPU, you can your GPU, you can access them access them access them much faster and much faster and much faster and get answers get answers get answers faster. You can faster. You can faster. You can route route route requests to models requests to models requests to models based on based on based on how how hot they are, or which hot they are, or which hot they are, or which model was model was last loaded and which was last unloaded. So, there are many So, there are many options for working options for working options for working with the with the with the latency parameter to latency parameter to latency parameter to determine the best determine the best determine the best models not only from a models not only from a models not only from a general perspective, general perspective, general perspective, but also from an but also from an but also from an infrastructure perspective. This is infrastructure perspective. This is infrastructure perspective. This is quite useful, and I quite useful, and I quite useful, and I think a lot of think a lot of think a lot of things will be things will be things will be implemented this way implemented this way implemented this way pretty pretty pretty soon. And yes, this is the idea that soon. And yes, this is the idea that soon. And yes, this is the idea that you can you can you can evaluate each model evaluate each model evaluate each model to determine which

to determine which to determine which ones are already "warmed up." ones are already "warmed up." In some cases, it In some cases, it is cheaper to go is cheaper to go is cheaper to go with a model that is already with a model that is already with a model that is already active, even if active, even if active, even if it is more expensive, than to it is more expensive, than to it is more expensive, than to reload a reload a reload a model that is cheaper model that is cheaper model that is cheaper but not loaded into but not loaded into but not loaded into your GPU. So this logic your GPU. So this logic your GPU. So this logic can be configured can be configured can be configured to optimize to optimize to optimize costs and costs and resource utilization at a level that resource utilization at a level that goes beyond the scope of goes beyond the scope of goes beyond the scope of individual tools individual tools . I mention this only to . I mention this only to show how show how powerful powerful powerful model inference and good model inference and good infrastructure tuning are for infrastructure tuning are for opening up new opening up new use cases. There is use cases. There is also potential to also potential to also potential to create create create solutions such as “judges” that solutions such as “judges” that solutions such as “judges” that communicate with communicate with communicate with each other, or “ each other, or “ each other, or “ model boards,” etc. model boards,” etc. model boards,” etc. Again, this is a question for Again, this is a question for Again, this is a question for the future, but it's the future, but it's the future, but it's another direction where you another direction where you another direction where you could, for example, could, for example, could, for example, use use reasoning tools or reasoning tools or adjust answers adjust answers adjust answers depending on the depending on the depending on the complexity of the tasks complexity of the tasks complexity of the tasks and what models and what models and what models are being called out. During are being called out. During are being called out. During development, I ran into a development, I ran into a development, I ran into a problem: for problem: for problem: for one of the one of the summarization tools, I summarization tools, I was using was using was using a model that had its a model that had its a model that had its own " own " own " reasoning tokens." Therefore, reasoning tokens." Therefore, reasoning tokens." Therefore, the summarization the summarization the summarization was either very was either very was either very short or did

short or did short or did not occur at all, not occur at all, not occur at all, since everything since everything since everything was performed at the was performed at the model reasoning stage. So, you model reasoning stage. So, you also need to also need to also need to be mindful of what be mindful of what be mindful of what models you models you models you are using and whether are using and whether are using and whether you are including you are including you are including thinking tokens. But sometimes thinking tokens. But sometimes thinking tokens. But sometimes considerations can considerations can considerations can be useful, and it's be useful, and it's be useful, and it's important to have a way to important to have a way to important to have a way to quickly quickly quickly implement it from the implement it from the implement it from the infrastructure side. And infrastructure side. And infrastructure side. And of course, there are some of course, there are some of course, there are some drawbacks to drawbacks to drawbacks to using using using a router, a board, a router, a board, a router, a board, or something similar. or something similar. However, we also However, we also pay a lot of pay a lot of pay a lot of attention to evaluation and attention to evaluation and attention to evaluation and try to try to try to test models test models test models before doing so. So before doing so. So before doing so. So you can implement you can implement you can implement different ideas here. And so, we different ideas here. And so, we different ideas here. And so, we don't really want to don't really want to don't really want to waste GPU resources waste GPU resources waste GPU resources on models that on models that on models that think forever and think forever and think forever and then nothing then nothing then nothing happens, or happens, or happens, or use use use biased models as biased models as biased models as judges, or judges, or judges, or optimize something that optimize something that optimize something that then...something that then...something that then...something that avoids the avoids the avoids the real goal, real goal, real goal, becoming just something becoming just something becoming just something that the models "hack" that the models "hack" that the models "hack" and you and you and you can never use can never use can never use it as a proper assessment.

it as a proper assessment. You can You can browse our browse our browse our repository. This is a QR code, repository. This is a QR code, repository. This is a QR code, if you haven't scanned it yet if you haven't scanned it yet if you haven't scanned it yet . I think the . I think the . I think the main takeaway main takeaway main takeaway today is that today is that today is that infrastructure and infrastructure and infrastructure and inference are much more inference are much more inference are much more important, even important, even important, even for very convenient API for very convenient API solutions where you're solutions where you're solutions where you're working with cloud working with cloud working with cloud code and might be code and might be code and might be handling handling handling sensitive data sensitive data sensitive data or specific or specific or specific tasks. Still, the tasks. Still, the tasks. Still, the basis remains basis remains basis remains inference. And inference. And inference. And for example, I'll for example, I'll for example, I'll go here again. This go here again. This go here again. This will also be will also be will also be published and you published and you published and you can find what can find what can find what files are supported, files are supported, files are supported, where it is appropriate where it is appropriate where it is appropriate to use LoRA, to use LoRA, to use LoRA, check out check out troubleshooting guides, and more.

Also, most importantly, don't forget to ask for don't forget to ask for don't forget to ask for your API keys and more. your API keys and more. We also have a complete We also have a complete model catalog that model catalog that model catalog that we are updating to we are updating to we are updating to make it make it make it more detailed and more detailed and more detailed and useful; you can useful; you can useful; you can choose different choose different choose different inference types and inference types and inference types and architectures architectures architectures because because they they need to be adapted need to be adapted need to be adapted into different packages to optimize models and inference. After all, into different packages to optimize models and inference. After all, into different packages to optimize models and inference. After all, all models can all models can all models can implement the implement the implement the attention mechanism in attention mechanism in attention mechanism in different ways. Therefore, you different ways. Therefore, you different ways. Therefore, you cannot run them and cannot run them and cannot run them and perform a perform a perform a forward pass in forward pass in forward pass in the same way. So the same way. So the same way. So different families of different families of different families of models require models require models require different different different runtimes to make everything runtimes to make everything runtimes to make everything work, and there are a work, and there are a work, and there are a lot of nuances here— lot of nuances here— models are models are models are tagged depending on tagged depending on tagged depending on whether they whether they whether they perform encoding perform encoding perform encoding or evaluation. So or evaluation. So or evaluation. So you can also you can also you can also use use use scoring models scoring models scoring models that do that do that do reranking, reranking, reranking, even in this even in this even in this case using case using tool calls, and tool calls, and the advantage is the advantage is the advantage is that you can that you can that you can essentially play with your essentially play with your essentially play with your own version of own version of own version of search through this MCP, and search through this MCP, and search through this MCP, and I would really appreciate I would really appreciate I would really appreciate if you could if you could if you could contribute to that or contribute to that or contribute to that or give us ideas on give us ideas on give us ideas on how to tweak it and how to tweak it and how to tweak it and build more build more build more useful things for people useful things for people useful things for people on top of that.

on top of that. We recently We recently ran a hackathon in ran a hackathon in ran a hackathon in London with Tech London with Tech London with Tech Europe and DeepMind, where a lot of Europe and DeepMind, where a lot of Europe and DeepMind, where a lot of people tested people tested people tested it out and were excited it out and were excited it out and were excited to work on both the to work on both the to work on both the search and search and search and reranking aspects, as well as the reranking aspects, as well as the reranking aspects, as well as the generation. So generation. So generation. So being able to do both being able to do both being able to do both with the with the with the support of this support of this support of this output is really cool output is really cool . And again, I . And again, I . And again, I encourage you encourage you encourage you to take advantage of these to take advantage of these to take advantage of these free free cluster infrastructure credits that you cluster infrastructure credits that you can use. can use. can use. We also have many We also have many We also have many examples and complete examples and complete examples and complete documentation for documentation for documentation for you to you to you to peruse. It's peruse. It's peruse. It's essentially a quick start and essentially a quick start and essentially a quick start and then different versions of the then different versions of the then different versions of the models and how you models and how you models and how you can look at can look at can look at migration, for example, migration, for example, migration, for example, if you have a if you have a if you have a workflow built workflow built workflow built using the OpenAI API, you using the OpenAI API, you using the OpenAI API, you say, "Okay, cool. I say, "Okay, cool. I say, "Okay, cool. I want to want to want to test this out quickly. What test this out quickly. What test this out quickly. What can I do?" You can I do?" You can I do?" You also have these options. And this is a also have these options. And this is a also have these options. And this is a repository. So, repository. So, repository. So, again, this is a package again, this is a package , and here it shows what , and here it shows what , and here it shows what it opens, and how the it opens, and how the it opens, and how the credentials are configured credentials are configured credentials are configured . So, . So, . So, you can view the readme here.

you can view the readme here. Here is an example. Here is an example. Also, if you add Also, if you add Also, if you add your examples and your examples and your examples and anything else, that would be anything else, that would be anything else, that would be very helpful. And that's very helpful. And that's where you can where you can apply for apply for apply for credits to credits to credits to help you help you help you build things and build things and build things and test them. This test them. This test them. This was very popular was very popular was very popular at the hackathon, so I would be at the hackathon, so I would be at the hackathon, so I would be very happy if very happy if very happy if you tried it. And I you tried it. And I you tried it. And I think as we think as we think as we slowly slowly slowly approach the approach the approach the end, I can also end, I can also end, I can also answer answer answer questions. You will just questions. You will just questions. You will just talk, and I talk, and I talk, and I will repeat will repeat will repeat the questions for the record. the questions for the record. So, yes, I'd be happy So, yes, I'd be happy to answer a few to answer a few to answer a few questions, and we can also questions, and we can also experiment with experiment with this more if you this more if you this more if you have a request you have a request you have a request you want to try. want to try. Yes.

Yes. So, using So, using So, using MCP through Claude, I'm still MCP through Claude, I'm still MCP through Claude, I'm still concerned about concerned about privacy issues, privacy issues, right? Because Claude right? Because Claude right? Because Claude will consume the will consume the will consume the tokens of any tokens of any tokens of any document you document you document you may send to may send to may send to MCP. How do you control MCP. How do you control MCP. How do you control the fact that Claude is the fact that Claude is the fact that Claude is still reviewing this data still reviewing this data still reviewing this data and sending it to MCP? and sending it to MCP? and sending it to MCP? Well, actually, he Well, actually, he Well, actually, he won't do that. So won't do that. So won't do that. So the problem is... the the problem is... the issue is about issue is about issue is about privacy. privacy. privacy. So, in essence, Claude So, in essence, Claude So, in essence, Claude will only receive the will only receive the will only receive the edited version of the edited version of the edited version of the document. So, it document. So, it document. So, it won't receive the won't receive the won't receive the full document full document full document you upload there. you upload there. This is the architecture we have This is the architecture we have now. So he now. So he now. So he will only see this will only see this will only see this part of the data. part of the data. part of the data. But I say, if you But I say, if you But I say, if you write in Claude, you give write in Claude, you give write in Claude, you give him a task, him a task, him a task, Yes. Yes. Yes. which will eventually which will eventually which will eventually end up in the MCP, end up in the MCP, end up in the MCP, right? That's what it's right? That's what it's right? That's what it's needed for. needed for. needed for. Yes. Yes. Yes. Claude will make Claude will make Claude will make the decision to call the MCP. the decision to call the MCP. the decision to call the MCP. At some point, he At some point, he At some point, he will hand over an will hand over an will hand over an edited version of edited version of edited version of the file as part of the file as part of the file as part of his his his decision-making process. decision-making process. Yes, that is, it is possible Yes, that is, it is possible to test whether to test whether to test whether this is happening or whether this is happening or whether this is happening or whether Claude is going beyond the Claude is going beyond the Claude is going beyond the rules. Of course, this rules. Of course, this rules. Of course, this can be tested more and can be tested more and can be tested more and more, but in more, but in more, but in my tests, if you my tests, if you my tests, if you host MCP and everything host MCP and everything host MCP and everything else in a virtual else in a virtual else in a virtual private cloud, it private cloud, it private cloud, it works without any works without any works without any data leakage. But I'm data leakage. But I'm data leakage. But I'm sure you can sure you can sure you can add more add more add more safeguards to safeguards to safeguards to strengthen this at the strengthen this at the strengthen this at the initial stage, initial stage, initial stage, or in the worst or in the worst or in the worst case you can case you can case you can say, "Okay, I'm

say, "Okay, I'm say, "Okay, I'm worried. I can worried. I can worried. I can create my own create my own create my own shell and shell and shell and use open-source, that's it use open-source, that's it use open-source, that's it . “Yes, yes. . “Yes, yes. . “Yes, yes. Yes, yes. And Yes, yes. And Yes, yes. And so, even...you so, even...you so, even...you can save can save can save tokens and get tokens and get tokens and get better context, better context, better context, even if it's, even if it's, even if it's, for example, Claude code, but for example, Claude code, but for example, Claude code, but it could be any it could be any it could be any shell. If you shell. If you shell. If you implement implement implement tool calls well tool calls well , I think it can be , I think it can be , I think it can be done, but I would be done, but I would be done, but I would be happy for input on happy for input on happy for input on this. It would be very this. It would be very this. It would be very useful to have this useful to have this useful to have this level of pre- level of pre- level of pre- processing. Yes, another one. processing. Yes, another one. processing. Yes, another one. Yes. Yes. Yes. Yes. Mhm. Yes. Yes. Yes. Yes. Mhm.

Yes. Yes. So, again Yes. Yes. So, again , the question is , the question is , the question is how do we set this up how do we set this up to ensure to ensure to ensure privacy, privacy, privacy, right? So on the other hand right? So on the other hand ...on the other hand, the ...on the other hand, the cluster in this cluster in this case—this is what case—this is what case—this is what we have, but it could we have, but it could we have, but it could be your own be your own be your own cluster. So, through cluster. So, through cluster. So, through the repository, you the repository, you the repository, you can set up your can set up your can set up your own cluster in own cluster in own cluster in your...Yes, that's right. your...Yes, that's right. your...Yes, that's right. Yes, you can Yes, you can Yes, you can configure the cluster configure the cluster configure the cluster however you want. I'm just however you want. I'm just however you want. I'm just using the using the using the managed cluster managed cluster managed cluster that we have for that we have for that we have for demos and demos and demos and examples, and we examples, and we examples, and we use it for things use it for things use it for things like like like hackathons and stuff. So hackathons and stuff. So hackathons and stuff. So we provide we provide we provide the equipment, but you the equipment, but you the equipment, but you can totally can totally can totally do it do it do it yourself. So it's a yourself. So it's a yourself. So it's a self-hosted self-hosted solution, and as I solution, and as I solution, and as I said, in the repository said, in the repository said, in the repository you can you can you can apply Terraform, apply Terraform, apply Terraform, configure all configure all configure all the infrastructure, and the infrastructure, and the infrastructure, and if you have if you have if you have GPUs and so on, you essentially GPUs and so on, you essentially configure all of that, and it's configure all of that, and it's completely yours.

completely yours. Excuse me, one more time? Is there a Excuse me, one more time? Is there a file in MCP? It simply file in MCP? It simply file in MCP? It simply sends it to the sends it to the sends it to the main worker, and main worker, and main worker, and the file the file the file is processed there. So there is is processed there. So there is is processed there. So there is something like that, yes. And yes something like that, yes. And yes , you can, I think, , you can, I think, , you can, I think, get get get free free free computing computing computing power at the entrance to power at the entrance to power at the entrance to try it out. Again try it out. Again try it out. Again , if you want to , if you want to , if you want to use your use your use your own hosting, you own hosting, you own hosting, you can do so. It's can do so. It's can do so. It's not, um, a strict not, um, a strict not, um, a strict restriction in that restriction in that restriction in that sense. We just sense. We just sense. We just do it to make things do it to make things do it to make things happen faster. happen faster. But again, you But again, you don't have to don't have to don't have to use our use our use our cluster. You can cluster. You can cluster. You can use your use your use your own hosting. And own hosting. And own hosting. And yes, you can also yes, you can also yes, you can also add me on LinkedIn. We add me on LinkedIn. We add me on LinkedIn. We can can can chat there and, chat there and, chat there and, um, thanks for coming.