WEBVTT 00:00.000 --> 00:15.520 Hello everyone, thank you, thanks for coming, we appreciate to have such a full room today, 00:15.520 --> 00:20.360 it's very nice, tells us that there's interest in similar kind of problems probably 00:20.360 --> 00:24.200 than the ones we are trying to solve. 00:24.200 --> 00:30.320 I hope your thoughts and experiences go in pretty well, and let's try to enjoy it. 00:30.320 --> 00:34.000 If you have questions, we're going to try to leave some time at the end. 00:34.000 --> 00:37.520 So yeah, let's dive in. 00:37.520 --> 00:41.520 Before I can talk about Hateron, I need to talk a little bit about Cares, because you 00:41.520 --> 00:46.720 need to understand the context of why these lessons and what is it exactly that we'll 00:46.720 --> 00:48.280 learn in that process. 00:48.280 --> 00:49.280 So what is Cares? 00:49.280 --> 00:57.000 Cares is, we can call it a framework or a toolkit that takes most known, we could say, 00:57.000 --> 01:03.800 limited distributions and transform them from their traditional OCI image into image-based 01:03.800 --> 01:11.240 immutable system or what we call nowadays immutable system or special property in systems. 01:11.240 --> 01:16.400 It's very nice, because we try to be open to as many distributions as we can cover, that 01:16.400 --> 01:20.880 means you can start with Ubuntu, with DVN, with open suci, and most of these, as long 01:20.880 --> 01:27.520 as there are some small requirements like running certain init system and such. 01:27.520 --> 01:34.400 And it is declarative, that means that you start with Dockerfiles, and it is, and container 01:34.400 --> 01:40.840 base, like I was mentioning, and you use Jamal for the configuration is very similar to 01:40.840 --> 01:47.320 clouding, it's like a subset of cloud init, which for us is a feature, because a lot 01:47.320 --> 01:53.080 of our workloads are for people running cloud native workloads, so that means they don't 01:53.080 --> 01:56.080 have to learn a new tool. 01:56.080 --> 02:02.480 And like I said, we run Kubernetes or traditional Linux workloads, and the main goal of 02:02.480 --> 02:08.600 the project is to simplify data operations, so the idea is that you get, for example, atomic 02:08.600 --> 02:13.880 AB upgrades, and if for some reason, one of the upgrades didn't work, you still have your 02:13.880 --> 02:20.520 passive image that your system running at the edge doesn't all the sudden stop working, 02:20.520 --> 02:25.360 but you have more resilience on those systems plus other features. 02:25.360 --> 02:31.200 And we are 100% open source, there's no hidden feature behind a paywall or anything like 02:31.200 --> 02:32.200 that. 02:32.200 --> 02:38.560 We are at the moment, CNCF sandbox project, looking forward to keep growing in the ecosystem. 02:38.560 --> 02:43.560 Who makes cars, we're basically five maintainers at the moment. 02:43.560 --> 02:49.840 We have 10 official contributors, plus all the other people that have come and share a 02:49.840 --> 02:52.960 little bit of their contributions as well. 02:52.960 --> 02:59.160 We have the moment to organizations backing us up, is Mirantis is backing up one 02:59.160 --> 03:06.640 maintainer, and Spectre Cloud is backing up four maintainers, and we have had three 03:06.640 --> 03:11.840 stakeholders until now, these people who came for a certain period of time, made sure 03:11.840 --> 03:17.240 that a certain feature that was useful for them was working, and then they didn't need 03:17.240 --> 03:19.440 to continue contributing there. 03:19.440 --> 03:25.600 Yeah, my name is Maura Morales, and I'm going to give now the mic to my colleague here. 03:25.600 --> 03:29.200 Hi, I'm Dimitris. 03:29.200 --> 03:36.000 So Tarros initially, we didn't think we would have to build a distro, because Tarros 03:36.000 --> 03:39.880 made an idea is that this distribution agnostic, so it takes the distribution of our choice 03:39.880 --> 03:44.000 and converts that to an immutable OS, and that's all the Tarros features on top, the 03:44.000 --> 03:49.920 two operations, it makes it immutable, you have maybe upgrades and all, and some features 03:49.920 --> 03:55.320 which cannot be put on top of all the distributions, and that's what we're going to discuss 03:55.320 --> 04:01.760 next, but the main differentiator compared to the other similar projects is that it's not 04:01.760 --> 04:05.560 a distro, so you can use your own distro, whatever you have, contracts with or you have 04:05.560 --> 04:09.080 familiar with and such. 04:09.080 --> 04:13.880 So like the previous slide already showed the distributions, we're trying to basen, but 04:13.880 --> 04:20.600 there on top you have flavors without Kubernetes, we have two distributions of Kubernetes, 04:20.600 --> 04:25.080 out of the box, k0s and k3s, and then you have different boards, different architectures 04:25.080 --> 04:33.240 which we support and finally, you have BIOS based device or EFI, so that creates a big matrix, 04:33.240 --> 04:37.080 you imagine all these combinations, you have to produce artifacts and we're talking images here 04:37.080 --> 04:42.280 even cloud images, and there will be artifacts, we end up with almost 400 artifacts on these 04:42.280 --> 04:45.880 to release, which is fine, that's what we're supposed to do, the one thing you can't 04:45.960 --> 04:51.240 do, because it cannot be automated, it is to support all the features on all the flavors, 04:51.240 --> 04:55.800 it this distribution makes its own decisions on what it wants to support, for example, 04:55.800 --> 05:00.840 some of them are based on openersy, some others are on system D, and that means, 05:00.840 --> 05:04.440 okay, you can actually bring all the features, but you have to implement them yourself, 05:04.440 --> 05:09.720 which is a very hard task, so we support what we can on what we can, that's why not every distribution 05:09.720 --> 05:12.120 gets the same features. 05:15.880 --> 05:22.200 Yeah, so of course you can imagine that after a couple of years of doing every release around 400 05:22.200 --> 05:28.360 artifacts, we start hitting the same walls every now and then, so there were some lessons 05:28.360 --> 05:34.600 learned along the way, the first one is that surface area matters, and just to give you an example 05:34.600 --> 05:39.560 of what we're trying to talk about here is that whenever there's a CVE and there's a patch from 05:39.560 --> 05:46.120 upstream, all of our users have to build our new image, or we build the image for them sometimes, 05:46.120 --> 05:53.720 but the point is that a process of image building has to happen, and then a process of upgrade. 05:53.720 --> 05:59.480 This might not be a problem if you're running in a data center where you have unlimited bandwidth, 05:59.480 --> 06:05.960 maybe it's an annoyance at most, but it's not a problem for you, but what happens when you're 06:05.960 --> 06:11.400 running at the edge, which is one of the main use cases that we have at Caires, well you might 06:11.400 --> 06:17.160 not have very good network and in those scenarios, you don't want to be upgrading that often, 06:17.160 --> 06:25.800 but you still want to remain secure, right? So what we analyze is that one of the main factors 06:25.800 --> 06:32.280 impacting us is the C library, so we looked at the different distributions, and we saw, for example, 06:32.280 --> 06:37.640 that I'll find, just to give you an example, at the moment that we were making these slides, 06:37.640 --> 06:44.600 had six CVE's found versus Gnucy library, which had 157. By the way, this is not 06:45.480 --> 06:50.040 comparison into saying what's good and what's bad, not at all. What we're trying to say here is 06:50.040 --> 06:58.120 the state of things. So that means that with muscle, we had to build less images less often. 06:58.680 --> 07:08.920 The other thing is that in terms of size, muscle is one fifth of the library, and that means 07:08.920 --> 07:15.160 that for the finalized image gets to be a lot smaller, and again, if you're at the edge, 07:15.160 --> 07:23.000 you want this smallest image possible. So okay, we said fantastically, let's do everything with 07:23.000 --> 07:30.600 Alpine only, right? But of course, that's not the truth. Yeah, so we based this on the assumption 07:30.600 --> 07:38.600 we have two microphones, we don't rotate so much. So yeah, like we said, the components you have 07:38.600 --> 07:43.400 that the features you can build on, so like we said before, some distributions are based on 07:43.400 --> 07:48.760 system, these, some others on OpenRC, one of the main features, and actually, if you're familiar 07:48.840 --> 07:54.440 with this blog post, let's not puttering is an implementation of that, I would say, 07:54.440 --> 08:00.280 is what we call trusted boot, and at least the middle part, the measured boot part is based on 08:00.280 --> 08:05.960 system D, so okay, you could implement it on OpenRC, I guess, if you want it, but it comes out of 08:05.960 --> 08:10.680 the box and we're trying to use the components as they come, we don't have the capacity to 08:10.680 --> 08:15.480 implement everything on everything, right? So one of the main features is based on that, that's 08:15.560 --> 08:20.920 why, for example, we have this feature on system-debut-based flavors, but as you see, 08:20.920 --> 08:29.080 Alpine is not one of them, so the idea we had, let's build everything on Alpine is not valid. 08:29.080 --> 08:33.320 So when this situation, this telelimatization, where you have three things, and you get to choose 08:33.320 --> 08:38.360 only two, so if you want it, it's more footprint, let's say, muscle for this scenario, and you 08:38.360 --> 08:43.880 wanted the Cairo's experience, I would say, yeah, you can get the Cairo'sified Alpine, 08:44.520 --> 08:51.080 you could get the measured boot feature with the Cairo's experience if you get one of the others, 08:51.080 --> 08:56.200 but we didn't have anything for the combination of small footprint and measured boot, or for all of 08:56.200 --> 09:02.040 them, if you could dare wish for that, but there is one project that actually did that at the 09:02.040 --> 09:06.200 same time we did that, and that's post-market OS, and we thought about going back and say, okay, 09:06.200 --> 09:12.280 why not Cairo's five post-market OS, but like we said, one project has different goals, 09:12.280 --> 09:18.120 it's very hard to align, so if post-market OS decides to make some changes or compile the 09:18.120 --> 09:23.560 kernel differently or whatever, yeah, I mean, we could basically, sorry, it's like yeah, 09:23.560 --> 09:29.880 but if you make some choices, it may break it for us, and it happened before that some 09:29.960 --> 09:36.360 patches they did for example in Ubuntu, Ubuntu was running fine on Raspberry Pi with Cairo's 09:36.360 --> 09:43.320 top, but at some point because they don't use Ubuntu runs Raspberry, Raspberry, they didn't care 09:43.320 --> 09:48.520 about the specific flag in the kernel compilation, which we cared about because we use Ubuntu, 09:48.520 --> 09:54.280 and it kind of broke Cairo's with Ubuntu on Raspberry, and other things like different versions, 09:54.360 --> 09:59.480 other versions of system D, and things like that, so you don't really control what you get, 09:59.480 --> 10:04.360 you get what you get, and it's better to stay after first and not have to patch things like we said. 10:05.800 --> 10:08.920 So yeah, I mean the next idea is obvious. 10:10.920 --> 10:15.240 Yeah, so we went back to the drawing board and we said, okay, what if we build it ourselves, 10:15.240 --> 10:20.680 right, like famous last words, how car can it be, it's just a Linux kernel, 10:21.640 --> 10:31.800 sorry, I'd say it does two words together, plus system D, and minus patches, turns out it's 10:31.800 --> 10:41.880 a lot harder than what it seems, but as we heard, there were some projects doing that, unfortunately 10:41.880 --> 10:47.800 we found out too late, now we finally got to meet here in Fossil, and maybe we got to collaborate 10:47.800 --> 10:53.640 in the future, but in the meantime, we decided, okay, let's build Haydron, what is Haydron? 10:53.640 --> 10:59.240 It's based, the core, as I was mentioning, is Mosul system D, and Vanilla Linux kernel, 10:59.240 --> 11:04.920 thanks to now the patches that have been contributed to system D with Mosul, that also works really well for us. 11:06.120 --> 11:11.160 For booting, you get two options, you either get the booting with trusted boot with EFI, 11:11.160 --> 11:17.160 or you get a traditional what we call standard boot with Dracut and Grub, and we try to stay as 11:17.160 --> 11:24.040 line to any of the projects we use as much as possible, so if they make a change, 11:24.920 --> 11:31.800 recently we were checking that the day after there is a patch upstream, we were already having it 11:31.800 --> 11:37.640 in our pipelines, which is really cool. But now if you notice, we spoke about the main components, 11:37.640 --> 11:43.160 and the main components doesn't include a package manager, so of course that is very small, 11:43.240 --> 11:49.320 which is very useful, but we still need to have ways in which we can upgrade our systems and so on, 11:49.320 --> 11:54.280 and that's the part that plays really well for us with Cairo, because we had already implemented 11:54.280 --> 12:00.520 all of that, so the two of them work together so that we can do the proper life cycle management, 12:00.520 --> 12:07.000 like we were doing with all the other distributions, so you again get all the AB atomic upgrades, 12:07.080 --> 12:14.120 you get the rollbacks, if something falls back, you get Kubernetes, if you need it to, 12:14.680 --> 12:20.360 we can do upgrades through Kubernetes and so on and so forth. So at the end of the day, 12:21.000 --> 12:29.880 we realize that our distribution is going to come and solve all the problems, that's actually 12:29.880 --> 12:36.520 not an option, but we realize that we need to still be using all of those distributions, 12:36.520 --> 12:41.240 but there was a small segment there in the middle where we needed small footprint, 12:41.240 --> 12:49.560 mesh of boot, and cloud life management that we didn't have before, and now with Hadron, 12:49.560 --> 12:53.880 we can cover that part, but whenever our users are going to come and say like, no, 12:55.000 --> 13:00.440 we have reasons to keep using, for example, Red Hat, or Susie, because we pay licenses, and we are 13:00.440 --> 13:06.840 in a, I don't know, we're going in government or whatever, then they're going to stay with that, 13:06.840 --> 13:12.520 we're going to continue providing the exact same support that we've been provided all of them until now 13:13.240 --> 13:19.000 for all of those distributions, the same thing if someone, because their team already had a certain 13:19.000 --> 13:23.400 know-how and they want to continue with the boom-to, they can keep doing that, but the difference is that 13:23.800 --> 13:30.920 we had cases where some people were coming to us, we're buying machines of the shelf, you know, 13:30.920 --> 13:37.400 nothing fancy, small nukes or whatever, and they come with old firmers that cannot boot 13:37.960 --> 13:45.880 200, more than a 200-meg image, then now we can provide through them a solution through Hadron. 13:45.880 --> 13:53.080 That's basically the reasons why we build it, the reason why we bring it here is because 13:53.080 --> 14:00.680 we think through the community is the best way to improve this work, so we want to get feedback from 14:00.680 --> 14:08.440 you guys, questions, if, like I was saying, we met with the people from Post-Mark market and we found out 14:08.440 --> 14:12.840 things we have in common, we would like to know more what we have in common, if we don't have to build 14:12.840 --> 14:18.120 the distribution ourselves, we will not do it, of course, but in this case, it still makes sense for us, 14:18.440 --> 14:27.720 but yeah, it's up to the community as well to reach out, like we're trying to reach other projects 14:27.720 --> 14:32.280 and figure out in which ways we can work together, and that's it, if there are any questions, 14:32.280 --> 14:34.280 we're happy to answer them, thank you. 14:48.440 --> 14:58.360 The question is, how are we distributed in Hadron? Yes, we right now build artifacts on GitHub 14:58.360 --> 15:02.840 and they are through their packages, you can download the OCI images. 15:06.840 --> 15:14.680 We also produce, whenever, so Hadron on its own might not be very useful in that sense, 15:14.760 --> 15:20.760 unless you want to build from scratch on your packages and all of that, the easiest is that 15:20.760 --> 15:27.960 the Cairo's releases will have already the full, yeah, the Cairo'sified version of Hadron 15:27.960 --> 15:33.720 and you can download it, like traditionally we were distributing Cairo's in GitHub as well, yeah. 15:34.600 --> 15:43.320 Yes? How much more? How much more is the Cairo's image that I would like to say? 15:46.520 --> 15:52.920 How much smaller is a Hadron image? That's a good and tricky question, because these are not 15:54.280 --> 15:59.800 sometimes people try to compare them just by speeding up a wound to container, for example, 15:59.800 --> 16:05.400 or any other container, and you have to think that in this case, when you have the Hadron 16:05.400 --> 16:11.880 containers, they already have everything necessary to be clarified, so it's not apples to apples, 16:11.880 --> 16:18.520 but more like an apple's word. But yeah, basically the smallest we have gone at the moment, 16:18.520 --> 16:25.960 I think it's 11 megs for an image, and once it's put on the system, of course, I think we're 16:26.040 --> 16:31.240 talking about a 145 megs, because you have to think it has to come with an active passive 16:31.240 --> 16:36.680 and recovery image altogether, so it only makes what I'm trying to say, sorry, that is really long, 16:36.680 --> 16:41.480 is that it only makes sense to compare it once it's in the system and compare it against other 16:41.480 --> 16:48.520 Cairo's systems, which before with Ubuntu used to be easily one gig or something like that, 16:48.520 --> 16:50.040 for us it's very significant. 16:50.360 --> 16:54.360 Yes? 16:54.360 --> 16:59.400 What's the preferred way to do that issue of software, like if you didn't do it just last year? 16:59.400 --> 17:03.480 Very good question. What's the preferred way to extend the Hadron? 17:04.600 --> 17:09.320 We're trying to base ourselves, like we like to say in the shoulders of giants, 17:10.120 --> 17:15.240 a lot of this work is based, for example, on system D, so we promote that people use 17:15.880 --> 17:22.520 system D, system extensions, that could be one way, why? Because we already provide the trusted 17:22.520 --> 17:27.240 mechanism, that means that you could also sign your extensions and make sure that everything in your 17:27.240 --> 17:34.040 stack is really only the software that you want to run there. But at the end of the day, 17:34.040 --> 17:38.600 is that Docker file, so you can do it any way you want, you can download binaries and copy them 17:38.680 --> 17:45.960 somewhere, or you can build them yourself within your Docker file. So it really depends on your needs 17:45.960 --> 17:48.200 and what's more practical for you. 17:56.200 --> 17:57.800 Thank you very much.