WEBVTT 00:00.000 --> 00:10.360 Okay, thank you, Ricardo, for warming up the stage. 00:10.360 --> 00:17.520 So, okay, this talk is about a stalwart and whether it's open source software, it's ready 00:17.520 --> 00:19.800 to run something like Gmail. 00:19.800 --> 00:27.320 So for those that, you know, no stalwart, it's an email and collaboration server that supports 00:27.320 --> 00:33.120 every single JMA protocol, they're except perhaps, yeah, a few things in that weapon, 00:33.120 --> 00:39.640 oh, baby, whatever, it's pronounced, but all the core ones, mail and so on are supported. 00:39.640 --> 00:56.280 And also support legacy protocols such as Carda, weather and double, okay, so the, so let's 00:56.280 --> 01:03.200 see what based on, of course, we don't know exactly what JMA, what Gmail is using to run 01:03.200 --> 01:11.280 out their scale, but we know a few things from research paper, research paper, so on. 01:11.280 --> 01:18.640 For example, we kind of know that for a span classification, they use vectorization, 01:18.640 --> 01:26.720 the molecule red pack, they use deep learning and ensemble models, then we know, do something 01:26.720 --> 01:33.600 called big-type or spanner for metadata, then colossus for the blobs and search, we don't 01:33.600 --> 01:40.880 know exactly how it's called, but yeah, we know it works really well. 01:40.880 --> 01:45.400 So how can we replicate this with open source? 01:45.400 --> 01:54.120 And here, let's talk a bit about the traditional email stack before store work. 01:54.120 --> 01:59.280 So the program with the traditional IMA servers such as DupCot for example, has a rely 01:59.280 --> 02:05.400 on mail there, mail is great for single server setups, but the program is that it does 02:05.400 --> 02:07.160 not scale. 02:07.160 --> 02:13.480 Some large mailbox provider can make it scale, but that leads to silos, for example, some 02:13.480 --> 02:18.880 solutions that use like the directors, where they keep the mailboxes in a compress, in a 02:18.880 --> 02:24.640 blob store, and when the user connects, the director, the one server becomes the owner of 02:24.640 --> 02:29.000 that mailbox, downloads it, uncompresses, and then all requests go to that node, and that 02:29.000 --> 02:36.440 node sends the update to the history store, which sounds very fragile. 02:36.440 --> 02:46.040 So then we have the traditional MTA, such as post fix, of course you can make them scale, 02:46.040 --> 02:50.040 but they have local cues, they're not, so you want certain node crashes, you need to 02:50.040 --> 02:55.760 recover your cue somehow, or the message is stuck there. 02:55.760 --> 03:02.320 Then we have the traditional span filters with the rely on bay base, I'm talking about 03:02.320 --> 03:05.680 the machine learning part, not all the rules and so on. 03:05.680 --> 03:13.480 So the single base has a decent accuracy, but you have to do a lookup per token, so that 03:13.480 --> 03:18.240 translates into one ready query, or whatever you're storing your tokens. 03:18.240 --> 03:23.840 And if you're using something like biograms or OSB, that multiplies by five, for example, 03:23.840 --> 03:33.080 and then if you keep global models and per user models, that multiplies by twice, so it's 03:33.080 --> 03:37.800 thousands of lookups per message, of course they can be cash, but still you have to do 03:37.800 --> 03:41.720 those lookups and then update, in the history of the environment, you have to distribute 03:41.720 --> 03:43.720 the updates. 03:43.720 --> 03:57.000 So still has been built from day one to be distributed, actually the version 0.1 has the 03:57.000 --> 04:05.560 rough consensus protocol built in, so it uses ROX-DB as a backend, but I was not so happy 04:05.560 --> 04:11.400 about having rate replicas, I want to reach node to be like a primary, to take right as 04:11.400 --> 04:16.920 well, that's why for this to be the environment that I decided to switch it to a foundation 04:16.920 --> 04:18.960 DB. 04:18.960 --> 04:27.160 So just like in a Gmail, Stowers separates a different amount of stores, this data store 04:27.160 --> 04:36.560 for stuff that changes often, such as flags, calendar contacts, indexes and so on, blobs 04:36.560 --> 04:46.200 are for static data such as raw messages and ship scripts and files from finals, then 04:46.200 --> 04:54.560 there's a full-text search engine for sending your messages or calendar contacts and so on 04:54.560 --> 05:00.160 and memory store for temporary data. 05:00.160 --> 05:06.000 So in Stowers, all nodes are equal-built, peers, so they're not really replicas, they're 05:06.000 --> 05:11.560 no leaders, no directors, users can connect to any node, no matter what protocol, I'm 05:11.560 --> 05:20.720 up, JMAP, tab, so you can scale or is simply by adding nodes and you can out-to-scale 05:20.720 --> 05:25.960 by running, running, throwing an orchestration platform. 05:25.960 --> 05:33.880 So this is how we would look like the sample design, so you have your client connecting 05:33.880 --> 05:39.120 to any node and then you have a different storage and a coordinator that we're going to discuss 05:39.120 --> 05:41.800 briefly later. 05:41.800 --> 05:48.800 So the data store, that's where the metadata store is recommended for very large environments 05:48.800 --> 05:56.880 is foundation to be, because it's used by Apple, for example, where I cloud mail is used 05:56.880 --> 06:00.440 in foundation to be. 06:00.440 --> 06:04.600 But then of course, we are not running something in Gmail's calendar and you need distribution, 06:04.640 --> 06:09.280 there is some posts around the internet, how can you say, yeah, I don't have to upgrade, 06:09.280 --> 06:13.880 I use postgres instead, same thing, foundation to be great, I use postgres instead, postgres 06:13.880 --> 06:18.200 also a great alternative to use, especially with read replicas. 06:18.200 --> 06:22.480 So if you don't need that scalability, postgres is also fine. 06:22.480 --> 06:28.480 In fact, Thunderbird is going to release a service called Thundermail using Stower and they're 06:28.480 --> 06:33.720 using postgres, so we'll see how well it goes for that. 06:33.720 --> 06:42.960 Then for the Blob store, the one without any staff, although this needs to be really tested 06:42.960 --> 06:49.160 in detail, because it will be slow with high throughput. 06:49.160 --> 06:53.960 But that is the thing that scales the most. 06:53.960 --> 07:01.200 I know that for self-hosted, if you're running, you also use a three compatible API, but 07:01.240 --> 07:04.960 it's important that they're not file, file sheet and dependencies, so they're no side 07:04.960 --> 07:06.600 of like in mail there. 07:06.600 --> 07:14.560 Then if the search, the full-text search, you can use, you can use the built-in search 07:14.560 --> 07:20.480 changing with Foundation TV, or you can use elastic search open search is Java is not an issue 07:20.480 --> 07:24.680 for Rust people, it is a bit sorry for the Java code. 07:24.680 --> 07:29.920 And mainly search also for smaller deployments. 07:29.920 --> 07:35.040 Then for the temporary data, you can use the radius, same cache, this is where the right 07:35.040 --> 07:41.880 limiting another log and temporary information is stored. 07:41.880 --> 07:46.800 Then in Stower, the MTAQs are totally distributed, so if one node crashes, the other one can 07:46.800 --> 07:53.920 pick up the pending task from that Q. So you can keep also, if you have a lot of messages 07:53.920 --> 07:59.320 pending for the delivery, you can start adding nodes to pick up nodes, to pick up 07:59.320 --> 08:02.440 jobs from that Q. 08:02.440 --> 08:09.880 Then for span classification, version 0.15, made a big improvement in performance and also 08:09.880 --> 08:15.600 classifying accuracy, classification accuracy by switching to the follow-the-regularized 08:15.600 --> 08:17.560 leader with optimal updates algorithm. 08:17.560 --> 08:22.920 That is, it's from Google Research, it's used in other words, it's an, for other 08:22.920 --> 08:29.920 words, click prediction, it's an online learning algorithm has also better accuracy that 08:29.920 --> 08:33.840 base, because base has assumed feature in the pendants, so to make it work, you need to use 08:33.840 --> 08:45.480 bi-gram or OSV, so FTRL is an optimizer function for a logistic regression model, so there 08:45.480 --> 08:51.640 are no lookups with this algorithm, with this design, sorry, not only algorithm, because 08:51.680 --> 08:56.880 is in the hash intrigue, the hash intrigue is another Yahoo paper, so this is U.P.R.O.E. 08:56.880 --> 09:02.520 still used nowadays by Yahoo, which is compressing the features of your model into a fixed 09:02.520 --> 09:08.000 size, and in order to reduce collisions, they use it, they change the sign with a second 09:08.000 --> 09:15.240 hashing function, and their paper shows that they can fit in 8 May of its own memory around 09:15.240 --> 09:24.000 2 to the power 20 features, the email for global and per user models for 430,000 users, 09:24.000 --> 09:29.880 or 4 million emails, or 3.5 million emails, so the software by default is as to the power 09:29.880 --> 09:35.280 20, but you can increase that to any number, and if you start having per and with collisions, 09:35.280 --> 09:41.840 you can use another paper that is called the improves on a hashing trick, and uses a 09:41.920 --> 09:50.760 kukuhashing, which is the kukuhashing algorithms, but updated, modifying the hashing trick, 09:50.760 --> 09:53.960 so to raise the collisions. 09:53.960 --> 10:00.400 So then in store where you have another component, which is a cluster coordination, so 10:00.400 --> 10:04.400 what it does is when you, for example, there is a J map out there, it is in one node, and 10:04.400 --> 10:10.120 the per hashing, the J map is written, or the user is connected with the 11th stream to 10:10.120 --> 10:15.080 another node, so those are the things to be distributed, so for that another kind of 10:15.080 --> 10:19.400 update, like failure, and so on are distributed with a coordination mechanism. 10:19.400 --> 10:26.080 If you don't want another dependency, you can use a clips, a CNO, or if not something like 10:26.080 --> 10:35.480 Apache Kafka or Red Panda, or something with a high throughput, then for a cluster 10:35.480 --> 10:41.600 orchestration, the difference is that, so the orchestration is running the multiple instances 10:41.600 --> 10:51.200 of stalwart in a cluster, and then when there is a, if you need to add a node and you 10:51.200 --> 10:59.920 know automatically, the Jogasterity mechanism can do that for you, also when the load is 10:59.920 --> 11:06.880 low, some nodes can be removed from the cluster, so the stalwart can run something like 11:06.880 --> 11:11.040 Kubernetes, Apache Messers, and so on. 11:11.040 --> 11:21.000 So I wanted to go, this separately, the cluster topologies, so let's say you have 1000, 1000 11:21.000 --> 11:28.600 nodes, how you distribute them, you can make all those 1000 nodes, this is just for the 11:28.600 --> 11:33.400 stalwart, not for the other service like Zeta, it's not so on. 11:33.400 --> 11:43.720 So you could have all nodes doing everything, JMA, PC and DPA, IMA, Zav and so on, or if 11:43.720 --> 11:49.880 not you can separate those devices, this, you really don't know the base on the, what 11:49.880 --> 11:54.600 they will be doing by, by a protocol, for example, you could have some nodes for JMA, 11:54.600 --> 12:02.840 but therefore IMA, PC and PC and so on, if not, you can distribute them, you can 12:02.840 --> 12:08.280 know a weighted allocation based on how popular that protocol is and put more nodes on 12:08.280 --> 12:16.480 that, on that sub cluster, say, and then another way to do it is pairing protocols, for 12:16.480 --> 12:25.760 example, similar protocols, SETP, IMA, being one group, then the JMA, and DAB, protocol, 12:25.760 --> 12:31.840 so HTTP, protocol in one side, and so on, and then you could also create a cluster based 12:31.840 --> 12:42.720 on geographic distribution, you can have different stalwart clusters by, by region, accessing 12:42.720 --> 12:55.520 the same stores, so to summarize, so usually you can start with unify and then move based 12:55.520 --> 13:07.280 on on your traffic patterns, so how close are we to run something like JMA with open 13:07.280 --> 13:13.760 source, so how close are all vertices and how close are the individual open source components. 13:15.760 --> 13:23.120 So the closest we have to a JMA scale is Foundation DB, because it's super scalable and it's 13:23.120 --> 13:31.120 used in production by Apple, of course Apple is not JMA scale, but it's also amazing millions 13:31.120 --> 13:38.880 and millions of users, also could be shared that eventually sub clusters of Foundation DB possible. 13:40.320 --> 13:50.080 So this is a, I mean, formal comparison based on public available info, so foundation with a 13:50.080 --> 13:57.680 closer we have to a big table and spanner, then for blobs, safe, also a good alternative, 13:57.760 --> 14:06.080 because of the, excuse myself, for example, by CR and HP and some other companies, 14:07.840 --> 14:15.920 they claim they can scale to petalize to XOS of data, we need to make sure how fast it is only, 14:15.920 --> 14:22.880 if you can handle high throughput, like a lot of rights, when you start receiving a large amount 14:22.880 --> 14:29.360 of emails at the same time, if a safe can handle that low or if that will create a bottleneck. 14:29.360 --> 14:43.040 So yet this is another informal comparison of the, of a safe and, of course, so the only issue we have 14:43.120 --> 14:52.880 is, in my opinion, it's a search store, this is, because Gmail is excellent, a search in a scale, 14:53.600 --> 14:59.760 we have, for example, a melee search which is also a rust, written in rust, but it's a single load 14:59.760 --> 15:07.120 and the distributed one, the distributed version is commercial, and also it is not possible to 15:07.120 --> 15:12.080 map every single jame up or I'm a query to a melee search query, because it's, it's time for 15:12.080 --> 15:20.560 user-facing applications, so some queries won't work as your user's expect, then we have a elastic 15:20.560 --> 15:28.800 search, open search, that the other program solutions, they also work well, but they want 15:29.200 --> 15:41.040 the lever, the scale needed for running something like Gmail, right? So, how can we get there, 15:41.040 --> 15:50.320 how can we get close to Gmail scale? So what we need is a truly distributed search engine, 15:51.040 --> 15:57.360 so one thing, we're exploring is a sixth store which is a rustful text search 15:57.360 --> 16:02.800 library, library that is faster than another library, than TV using rust, so a competition 16:02.800 --> 16:07.760 who has a faster library and they like outperforming sea libraries and stuff like that, 16:08.320 --> 16:16.880 and this one is quite promising, and they have plans to use, from addition to be as a 16:17.440 --> 16:24.320 well, I propose that and they're open to the idea, so we'll see how that goes, because that will be 16:24.320 --> 16:33.520 also, we'll make a search really scalable. So for the spam classifier, we're also, we're almost 16:33.520 --> 16:40.160 there, for performance, the current model used by so always, it's quite similar to what Yahoo uses, 16:41.120 --> 16:49.120 the algorithm FtRL is used by Google, but for something else, for other words, 16:50.400 --> 16:58.800 the problem or spending things to solve is accuracy being as accurate as Google. 17:00.640 --> 17:06.640 For example, while working on the new classifier, I did some tests using redback, which is used 17:06.640 --> 17:11.040 by Gmail, this went off for sure, so it's a vectorization model, taller and too attacks, 17:11.920 --> 17:20.720 and to train a Kaggle dataset for spam filtering to like two hours on an M2 Max and a Macbook, 17:20.720 --> 17:29.040 while the FtRL proxy algorithm did that in seconds, and the accuracy improvement was 0.3% 17:29.120 --> 17:37.280 which for the deep learning mode. So for Google scale, that is millions of emails, but for 17:37.280 --> 17:45.040 the small guy that's not that much, it doesn't justify. So if we want to reach Gmail's 17:45.040 --> 17:51.440 accuracy, we need to use something that understands content, the intent of the email, not just 17:51.440 --> 18:00.480 counting words like linear models, are doing. So the goal is to match email level accuracy and 18:00.480 --> 18:08.240 efficiency. So so we're already supports integrating LLMs. You can connect to Open API, or 18:10.480 --> 18:18.160 30PT, or Cloud, whatever, or run your own model locally. But of course, the slow, it's slow and 18:18.160 --> 18:23.760 expensive. So we need something in the middle between linear models, something like the linear regression 18:23.760 --> 18:30.240 models and LLMs. And one potential alternative, while the bird models that could be combined with 18:30.240 --> 18:39.360 redback, and that would be great to understand. So these models can assist the linear models, 18:39.360 --> 18:46.640 not replace, but as more info, to decide if an email is spam or not. 18:48.960 --> 18:57.600 Then for what is clustering, store work has been tested in the lab, of course, with a setup 18:57.600 --> 19:02.880 with one million users. It was done done on a 10 node cluster, but of course, it's a lab testing. 19:02.880 --> 19:12.960 But so far, all the work in store work was implementing new protocols and new features, 19:12.960 --> 19:19.200 but now store work is feature complete. So from now on, the focus is performance and scalability. 19:21.520 --> 19:26.320 So yeah, so from now on, it's all the fine tuning comes. 19:27.040 --> 19:40.880 So to get to Gmail scale, after version 0.6, 0.16 is released, the idea is to use some 19:40.880 --> 19:49.200 credits, a world by GitHub, to create a very large cluster to test if possible, something like 19:49.200 --> 19:54.720 Gmail. So as use and burn those credits to simulate a Gmail deployment, of course, before 19:54.880 --> 20:02.720 something smaller will be attempted, right? And then go from there and to see how far we can push it. 20:04.640 --> 20:12.640 So the roadmap for store work, as mentioned before, is now store work feature complete. 20:13.520 --> 20:18.960 However, there are a few pending user enhancement, a user request and enhancements. 20:19.600 --> 20:26.800 Versions 0.16 is under development. It will no longer use tunnel for settings. 20:27.600 --> 20:33.600 It will be configured with JMAP with a new future preferences protocol, several settings, 20:35.280 --> 20:44.560 protocol that will be hopefully standardized at some point. So yeah, so the goal is to make store 20:44.640 --> 20:52.960 where a scale was as Gmail is an ambitious goal, but even if that means writing a store work 20:52.960 --> 21:03.120 search engine to rich.go, so that is the plan. So well, this is a current summary, 21:03.120 --> 21:12.560 basically of what we've been talking about. I believe we're like almost there, basically because 21:12.640 --> 21:20.880 of all the great technology of their foundation, the VNSA-1, but there's a lot of work pending, okay? 21:20.960 --> 21:22.960 All right, thank you very much. 21:32.000 --> 21:34.000 Questions? 21:36.320 --> 21:41.520 First of all, you're standing, yeah, sorry, it's all and how much 21:41.600 --> 21:46.800 localization do you recommend to someone for action? 21:49.840 --> 21:57.280 Okay, the question is how large an organization that recommends store work to be used or 21:58.080 --> 22:00.080 well 22:17.360 --> 22:20.800 Oh, if he's taking in production, why is the largest deployment? 22:21.120 --> 22:31.360 The largest I know is a 300,000, but they do mostly outbound. Then there are smaller 22:31.360 --> 22:40.720 requirements of 50,000, for example, in Germany, there's another 30,000 in France, and also the 22:40.720 --> 22:45.600 search engine khagi, I don't know if you know them, like it's a paid search engine, it's also using 22:45.680 --> 22:53.280 a store work, but I don't know exactly the one that are not, some users may be using it for 22:53.280 --> 22:57.680 with more mailboxes, but if they're not sharing that, if they're using the open source version, 22:57.680 --> 23:00.400 we'll never know. Those are the ones I know, yeah. 23:16.560 --> 23:20.560 Can you repeat how large I think? 23:22.960 --> 23:27.600 Oh, how large is a standard my expected to become? I don't know, we need to ask 23:28.400 --> 23:35.680 the standard people, but they're running our beta, and they haven't shared the numbers, 23:35.680 --> 23:41.920 but if my guess is 100, because they're hundreds of thousands of users based on the donations that 23:42.000 --> 23:47.920 receive, I'm sure it will be reached at level, and also because I think there will be 23:47.920 --> 23:53.920 also offering free email, free email accounts, so that will also increment the number. 24:01.360 --> 24:05.360 Oh, because email is evil. Why competing with? 24:05.360 --> 24:14.640 The Gmail also is you don't download your data, they're using it probably to train models, 24:15.360 --> 24:18.560 and so it's not complete because Gmail is also a service, this is a product, 24:19.280 --> 24:25.200 I probably run out your server, so you get to control of your data, so instead of hosting your email, 24:25.200 --> 24:33.680 Gmail, you can host it at your own server, and yeah, and I'll prevent them from using it to train models. 24:34.640 --> 24:41.920 That would be a good reason to release a stalwart to release stalwart. 24:45.520 --> 24:49.120 Yeah, yeah, it is, it is out there, it is on GitHub. 24:54.720 --> 24:56.560 You rest your happy phone now, okay? 24:56.560 --> 25:02.880 I'm sorry, you may provide their season still worked in Europe or in the United Kingdom, 25:02.880 --> 25:07.600 like here for commercial suffocates. 25:07.600 --> 25:11.840 I don't know, he's asking is there any mailbox provider, you know, the kingdom? 25:13.840 --> 25:25.920 Yes, I know one in the Netherlands, a super space, I think it's called, yeah, yeah, yeah, yeah, 25:25.920 --> 25:31.520 it's great. If you rewrite the writing a search, and you can also replace Google, 25:32.880 --> 25:34.320 yeah, that would be the most. 25:36.320 --> 25:43.520 I'm wondering about your benchmark in size, because for me, it's like a bit of an aberration, 25:43.520 --> 25:51.040 I have an integrated protocol, but then everybody's using the segments, so I want that benchmark, 25:51.040 --> 25:59.120 and I would love to be, to see how easy it can be with one of the instances 25:59.120 --> 26:07.040 of multiple users, how easy it is to set up, how easy it is to set up, 26:07.040 --> 26:17.840 how to verify the instances, how easy it is to set individual instances, compared to something very large, 26:17.920 --> 26:23.040 is that correct? Yeah, that is a main, that's what Stau is already doing really well, 26:23.040 --> 26:28.320 which is running like very easy to deploy, you can handle your DNS records, 26:29.120 --> 26:35.040 was so far only the TLS, the Acme records, but versions of 16 will handle your decim, 26:35.040 --> 26:40.240 everything in a related, so it's pretty simple to set up a small instance. 26:40.240 --> 26:42.800 That's one final question, the background, sorry. 26:43.040 --> 26:49.840 We use all work, which is fantastic, the one mission, Stau works, we build by Stau work, 26:49.840 --> 26:58.400 and that's right. Yeah, that's correct, because before it was in the UK, 26:58.960 --> 27:09.120 and until the EUing company is type of company is available, it's currently been migrated to the EU. 27:09.200 --> 27:14.720 Also, it's going to be brought to Europe. Yeah, it's already managed from Europe, 27:14.720 --> 27:20.640 it's the LLCs for investment from the US, only, but it's not, there's no control of the US. 27:20.640 --> 27:24.880 One of the key levels of sovereignty is that it's been built by a European country, 27:26.080 --> 27:31.520 whether European business, located in Europe or so, and why only LLC might get brought to some 27:31.520 --> 27:38.240 country, but in your movement, Europe, that's great. Yeah, yeah, that is a plan, it's already, 27:38.320 --> 27:43.920 there's already a UKLTD managing, but the idea after the Brexit, 27:44.480 --> 27:49.040 there was a problem, the company is slowly winding down, so this is a temporary structure, 27:49.040 --> 27:54.880 until the EU cooperation is available. Yeah. Okay, thank you very much.