WEBVTT 00:00.000 --> 00:22.160 OK, OK. 00:22.160 --> 00:26.620 OK. 00:26.620 --> 00:56.560 Okay, we're going to go ahead and get started here then. 00:56.560 --> 01:04.560 It's going to be done by Robyn and his team is relying on more transparent trustworthy 01:04.560 --> 01:06.560 sources for arts-linic packages. 01:06.560 --> 01:11.560 So let's give him our new attention. 01:11.560 --> 01:18.560 Hello everyone, thanks for attending this talk. 01:18.560 --> 01:23.560 So yeah, today we're going to talk about the decision that we took at Archenix to rely on 01:23.560 --> 01:28.560 what we'd qualify as more transparent and trustworthy sources for our packages. 01:28.560 --> 01:34.560 This is basically our response to the supply chain attack that's on Linux distributions, 01:34.560 --> 01:37.560 suffered from recently. 01:37.560 --> 01:40.560 Before we dive into it, allow me to introduce myself. 01:40.560 --> 01:41.560 My name is Robyn. 01:41.560 --> 01:44.560 I'm also known as Antis on the Internet. 01:44.560 --> 01:49.560 I am a Linux system and DevOps engineer by day and an open source contributor by night. 01:49.560 --> 01:54.560 I am primarily involved in Archenix in case it wasn't obvious yet. 01:54.560 --> 02:04.560 But I also contribute to reproducible builds and a bit to Alpine Linux as well. 02:04.560 --> 02:06.560 So let's start with a few definition. 02:06.560 --> 02:09.560 First of all, what's the supply chain? 02:09.560 --> 02:13.560 Of course, the supply chain is a very generic term that can apply to a lot of things. 02:13.560 --> 02:17.560 But I came up with a simple definition for the Linux packaging ecosystem. 02:17.560 --> 02:20.560 This is what we're talking about right now. 02:20.560 --> 02:26.560 So the Linux packaging supply chain is composed of all the elements that are involved in building 02:26.560 --> 02:29.560 and delivering packages to users. 02:29.560 --> 02:34.560 Whether it is, you know, technical components such as our build script, et cetera, 02:34.560 --> 02:37.560 but also the infrastructure the package is built on. 02:37.560 --> 02:41.560 The infrastructure that is used to deliver the package to users. 02:41.560 --> 02:49.560 And more importantly, also the individuals like included into this whole process. 02:49.560 --> 02:52.560 So what's a supply chain attack then? 02:52.560 --> 02:57.560 Well, a supply chain attack is when one or more of those components is compromised, 02:57.560 --> 03:03.560 leading in like undesired and unexpected behavior and malicious code being introduced 03:03.560 --> 03:06.560 in otherwise trusted packages. 03:06.560 --> 03:14.560 Now, do we have protections against supply chain attack? 03:14.560 --> 03:15.560 Yes. 03:15.560 --> 03:24.560 Fortunately, we have some mechanism to prevent or at least detect such supply chain attack. 03:24.560 --> 03:29.560 But monitoring and securing the entire supply chains remains a challenge, 03:29.560 --> 03:33.560 specifically the specificity of very early part of the supply chain. 03:33.560 --> 03:37.560 But we're going to see that later. 03:37.560 --> 03:42.560 So let's go through a quick list of the few of the mechanisms that we have to protect 03:42.560 --> 03:44.560 ourselves against supply chain attacks. 03:44.560 --> 03:46.560 First of all, we have checksums. 03:46.560 --> 03:51.560 So when we download the source for a package, we generate like a hash of the source. 03:51.560 --> 03:57.560 And then this hash is locked into our build script as the hash that we expect 03:57.560 --> 04:02.560 the next time we're going to download or use this source. 04:02.560 --> 04:06.560 So that ensures that the sources have not been returned in between multiple downloads 04:06.560 --> 04:09.560 or rebuilds of the package. 04:09.560 --> 04:13.560 But, unmatched checksum could indicate a supply chain attack, 04:13.560 --> 04:16.560 but it may also have different reasons. 04:16.560 --> 04:22.560 We have a few upstream that have this bad habits of recreating a release, 04:22.560 --> 04:25.560 like they say they made a typo in the man page. 04:25.560 --> 04:28.560 And instead of cutting a new point one release, 04:28.560 --> 04:31.560 they will discrificly like delete this one and recreate it. 04:31.560 --> 04:35.560 Please don't do that, guys. 04:35.560 --> 04:39.560 Or it can be get hub that decides to change. 04:39.560 --> 04:44.560 There are algorithm and get archive that will change every text sum of every get-up 04:44.560 --> 04:46.560 total that happened. 04:46.560 --> 04:50.560 They reverted it, but once that happened. 04:50.560 --> 04:57.560 And as I said, this is technically more of a locking mechanism rather than an actual integrity check. 04:57.560 --> 05:00.560 Once we generate the hash, we expect this to never change. 05:00.560 --> 05:02.560 And we lock the source that way. 05:02.560 --> 05:08.560 But it's not like it guarantees anything in the content of the source in the first place. 05:08.560 --> 05:11.560 Then we have the reproducible bills. 05:11.560 --> 05:18.560 Reproducible bills defines that from the same source with the same build option and the same build environment. 05:18.560 --> 05:24.560 Anyone should be able to reproduce the same artifact bit for bit identical. 05:24.560 --> 05:29.560 Cool thing with the reproducible bills is that it allows to independently verify the path from source to 05:29.560 --> 05:34.560 binary code, entering that no supply chain attack occurred in between. 05:34.560 --> 05:37.560 So for instance, I has a package maintainer. 05:37.560 --> 05:44.560 I could technically inject the malicious patch into my package before the delivery to user. 05:44.560 --> 05:47.560 Of course, all of this is very theoretical, right? 05:47.560 --> 05:50.560 I will not do that, but I technically could. 05:50.560 --> 05:58.560 And reproducible bills is a way to verify that I did not abuse my position because you could try to reproduce the package. 05:58.560 --> 06:04.560 And you should arrive at the same exact result as the package I am delivering to you. 06:04.560 --> 06:08.560 And if you don't, then I may have do something that's thing. 06:08.560 --> 06:14.560 The other cool thing with reproducible bills is that it gives you a technical verification process. 06:14.560 --> 06:17.560 So you don't have to trust me as an individual. 06:17.560 --> 06:19.560 You don't have to put your trust in me. 06:19.560 --> 06:27.560 You can actually verify it yourself with a technical process, which is obviously more difficult to compromise than an individual. 06:28.560 --> 06:32.560 While it's always a desired outcome for distribution reproducible bills, 06:32.560 --> 06:35.560 doesn't always cover the whole supply chain, though. 06:35.560 --> 06:37.560 So that's a bit of the limit of reproducible bills. 06:37.560 --> 06:43.560 Specifically the very early part, as I said, that I'm going to demonstrate that in a minute. 06:43.560 --> 06:54.560 And the last mechanism we can go over is cryptographical signatures with like open PGP keys, like new PG signatures. 06:54.560 --> 07:00.560 This hack does an additional integrity check by guaranteeing that an artifact has not been tempered with. 07:00.560 --> 07:04.560 Since it was signed, I think this is an important part here. 07:04.560 --> 07:11.560 So when you sign an artifact, if it gets tempered with, then the signature verification will fail. 07:11.560 --> 07:15.560 So this is an additional integrity check. 07:15.560 --> 07:21.560 And the other cool thing with signatures is that it allows for the authentication of an identity tied to the signer. 07:21.560 --> 07:25.560 So it's basically someone to blame if something that happens. 07:25.560 --> 07:31.560 So that's cool to have someone that is somewhat responsible for whatever they are delivering. 07:31.560 --> 07:37.560 But set identity may or may not relate to an actual person. 07:37.560 --> 07:44.560 Like an artifact could be signed by an identity that relates to the release team. 07:44.560 --> 07:46.560 Like the arsenics release team. 07:46.560 --> 07:48.560 And you don't really know who that is, right? 07:49.560 --> 07:53.560 So it will not always tie to an actual person. 07:53.560 --> 07:57.560 And it may or may not correlate with the creator of the artifact. 07:57.560 --> 08:02.560 The person that signed the artifact is not necessarily the person that created it in the first place. 08:02.560 --> 08:08.560 And there's no way to establish a natural connection between the creator of the artifact and the signature. 08:08.560 --> 08:13.560 We just assume that's the case, but it may not be. 08:13.560 --> 08:17.560 So I'm not going to spend too much time on this, but realistically signature are only relevant. 08:17.560 --> 08:23.560 If they are accompanied by a trust path, basically if you maintain a chain of trust between the different keys you trust. 08:23.560 --> 08:31.560 If there are multiple keys involved in signing artifact and you just blindly had new keys to your hearing, then you're doing it wrong. 08:31.560 --> 08:33.560 It basically defeats the purpose. 08:33.560 --> 08:40.560 You want to make sure that if you trust that key and a new key is involved, you have a path between the first key you trust and the second one. 08:40.560 --> 08:42.560 Creating the chain of trust that I'm talking about. 08:42.560 --> 08:49.560 If you don't have this bridge, then it defeats the purpose and you might as well just not verify signature, basically. 08:49.560 --> 08:53.560 So all of this to say, we have mechanism. 08:53.560 --> 09:02.560 And yet, you, and yet, this happened. 09:03.560 --> 09:13.560 So just a quick context here. So XE is a library to to compress archive, basically. 09:13.560 --> 09:23.560 And it was backdoor targeting SSH, which would theoretically allow any untrusted third party to access to your mission remotely. 09:23.560 --> 09:29.560 And this happened, this by the upstream package source been locked behind checksum. 09:29.560 --> 09:32.560 Being cryptographically signed as well. 09:32.560 --> 09:36.560 And this by the infected package being reproducible as well. 09:36.560 --> 09:43.560 So the three mechanism I showed you were correctly applied here, and yet the package got backdoor. 09:43.560 --> 09:45.560 So how did that happen? 09:45.560 --> 09:48.560 Well, there are a few things to note here. 09:48.560 --> 09:55.560 First of all, the malicious code was introduced in the upstream source by one of the upstream maintainer themselves. 09:55.560 --> 10:03.560 Which, as I said, is a bit difficult for us to verify because such early part of the supply chain are out of our control. 10:03.560 --> 10:11.560 It's very difficult for us as downstream or distributor to have some verification mechanism on the very upstream part. 10:11.560 --> 10:15.560 So this makes it very difficult for us to detect. 10:15.560 --> 10:22.560 And another thing that is important to note here is that part of the malicious code, actually the trigger for it, 10:22.560 --> 10:28.560 was obfuscated by only being present in the custom-made release table. 10:28.560 --> 10:38.560 So what I talk about as custom-made table is those distribution tables that are distributed as an extra artifact of the release, 10:38.560 --> 10:42.560 all the table that you download from upstream website. 10:42.560 --> 10:47.560 And a lot of packages build their packages against that source. 10:47.560 --> 10:49.560 And yeah, it can be abused. 10:49.560 --> 10:50.560 This is what happened here. 10:50.560 --> 10:57.560 So if you look at the raw sources and the git repository, there was no trace of that trigger for the malicious code. 10:57.560 --> 11:03.560 But if you download that extra tarble, then the malicious code was there. 11:03.560 --> 11:10.560 So that implies that there are different types of sources that we can use for our packages. 11:11.560 --> 11:14.560 I'm going to name a few, not all of them. 11:14.560 --> 11:17.560 But the first obvious one is plain vcs object. 11:17.560 --> 11:23.560 So just check out of the git commit or a git tag, basically a clone of the repo. 11:23.560 --> 11:26.560 You also have auto-generated table, but git force. 11:26.560 --> 11:33.560 So that is the GitHub, the auto-generated table, or the git lab one, or the code, but one, whatever you use. 11:33.560 --> 11:39.560 And you have those in-famous custom-made release table. 11:39.560 --> 11:45.560 So, oh, well, jumped a bit too far here. 11:45.560 --> 11:49.560 Why do we need custom-made release table in the first place? 11:49.560 --> 11:51.560 Why do they exist? 11:51.560 --> 11:58.560 Well, they are generally easier and most straightforward to work with on the packaging point of view. 11:58.560 --> 12:05.560 Because there are some upstream sources that are based on technologies that require 12:05.560 --> 12:10.560 preliminary step before you're able to work with it on the packaging side. 12:10.560 --> 12:17.560 Now, if there are packages in the room, I may raise some PTSD for you guys, but stuff like auto tools. 12:17.560 --> 12:20.560 Hit sub-modules, translation file, manpages. 12:20.560 --> 12:30.560 So, yeah, auto tools, for instance, needs this very opaque auto-gen bootstrap script stuff that you have to run before you're able to work with sources. 12:30.560 --> 12:36.560 Git sub-modules are used to pull third-party dependencies that you may or may not need. 12:36.560 --> 12:41.560 And you would need to do that before you're able to package your sources. 12:41.560 --> 12:45.560 Translation files may need to be generated, manpages as well. 12:45.560 --> 12:49.560 And usually what upstream does is that it gives you this custom-made table. 12:49.560 --> 12:52.560 Well, all of these things are already done. 12:52.560 --> 12:55.560 So you don't have to take care of that on your side. 12:55.560 --> 13:09.560 There are also historically signed more often than other source type, probably as a way for upstream to promote this type of sources against other types. 13:09.560 --> 13:18.560 But there are also very opaque by nature and more difficult and involved to audit, as compared to a Git repo that is publicly readable. 13:18.560 --> 13:27.560 So, in my opinion, historically we have put way too much trust on signature over other mechanisms. 13:27.560 --> 13:30.560 What will come back to that later? 13:30.560 --> 13:35.560 And this is, as opposed to, quote, unquote, transparent sources. 13:35.560 --> 13:42.560 So for us, this is like plain VCS objects or just a check out of a specific Git tag or specific commit. 13:42.560 --> 13:52.560 Or assuming that you trust those platform, we also decided that Git Hub Auto Generatorable or Git Lab Auto Generatorable are okay. 13:52.560 --> 14:02.560 Given that we trust those platform, but you know, all our code is hosted on GitLab, so hopefully we trust GitLab. 14:02.560 --> 14:08.560 And those sources are publicly readable, easily accessible, and therefore generally more audited. 14:08.560 --> 14:16.560 There are way more pair of eyes on a Git repository rather than on the content of those custom made toggle, right? 14:16.560 --> 14:20.560 It gives you this feeling of what you see is what you get. 14:20.560 --> 14:32.560 So, this is an oversimplified image, but I usually like to see transparent sources as a transparent box where I can see the content without having to open that box. 14:32.560 --> 14:35.560 And the custom made release toggle would be in the pack box. 14:35.560 --> 14:43.560 And the signature on top of the custom made toggle would basically just be a sticker on it that says trust me it's fine, you know. 14:43.560 --> 14:48.560 And I don't want to trust you, actually, I don't want to trust an individual. 14:48.560 --> 14:53.560 So, and you kind of have to do that with, would like custom made toggle. 14:53.560 --> 14:58.560 Our transparent sources, I can just look at it and say, okay, this is what I'm going to get. 14:58.560 --> 15:08.560 Now, I've said it may involve some eventual cost in terms of complexity into packaging because stuff like auto gen fraud at all, 15:08.560 --> 15:10.560 gets some modules, et cetera. 15:10.560 --> 15:13.560 Those will be on our side, we will need to do that. 15:13.560 --> 15:27.560 But we now think that it is worth, it is worth the effort, because distributing packages based on more transparent sources is generally a good move both for us and for our users. 15:27.560 --> 15:42.560 And as a matter of fact, you probably cannot see it clearly here, but as a matter of fact, this is the fix that we applied on the XE infected package and the fix is mostly just to switch to get sources rather than this custom made toggle. 15:42.560 --> 15:54.560 And you can see that we had to, whereas it's here, we had to have this auto gen step that we did not have to do beforehand, thanks to the custom made toggle. 15:54.560 --> 16:08.560 Now we just do it, and that's fine, honestly, and we have a, it's not like a bulletproof source, but at least it is more difficult to obfuscate malicious stuff in those source. 16:08.560 --> 16:12.560 So this is what we conclude from that study. 16:12.560 --> 16:18.560 Basically, so there is a link here, you have a QR code at the end to get the slide, so you can follow the link. 16:18.560 --> 16:31.560 The link is basically a package I switched from custom made toggle to get toggle and it implies like, I don't know, 15 gets submodules dependency, so the package went from this to this, basically, in terms of a build script. 16:31.560 --> 16:44.560 But it's still, it's still with the effort, because even if the cost of more complexity, we now think that those, quote unquote, transparent sources and more trustworthy sources should generally be considered a preferred source for our packages. 16:44.560 --> 16:51.560 We're got in cryptographical signatures, so like GPG signatures, basically. 16:51.560 --> 16:59.560 We only think that they are really relevant if accompanied by a trust path, so basically maintaining a chain of trust. 16:59.560 --> 17:07.560 And if you are not able to maintain that chain of trust, you might as well drop key verification. 17:07.560 --> 17:21.560 And if a choice has to be made between the two, so if you have to choose between transparent source or opax source but signed, we now think that transverse sources are equating signatures. 17:21.560 --> 17:28.560 Of course, in an ideal well, we would have both, both transparent sources and sign sources. 17:28.560 --> 17:38.560 So we encourage now our package maintainer to establish communication with upstream, basically telling us our expectation on that front. 17:38.560 --> 17:44.560 And it should be very easy to do, basically signing get tags is just what we need. 17:44.560 --> 17:51.560 So we can base our package against your get tags and have that signature as well. 17:51.560 --> 17:57.560 Maintaining a trust path for us should be very easy as well, it could be a signed commit, it could be a signed message. 17:57.560 --> 18:13.560 One thing that upstream usually does is having a maintainer.md file into the repo, and you would have like the key fingerprint of every key that are expected to sign releases. 18:13.560 --> 18:21.560 And just having signed commits, it's time you had a new key or remove a key from there is enough for us to have a chain of trust. 18:21.560 --> 18:36.560 So yeah, that should be an easy step for upstream to do and we are, we are now establishing communication with them to hopefully being an ideal well of transparent sources and sign sources. 18:36.560 --> 18:54.560 There are a bit extra resources about all of what I just said, so you have the XC, Archenex Security Advisory, which says again that part of the malicious code was confiscated into the custom made to our bowl and that the fix was for us to switch to the get tags instead. 18:54.560 --> 19:07.560 You have this RFC that I can jointly wrote with David Runger, which is another Archenex maintainer, and this is basically what I just said in that talk, but in a very very long document with a lot of technical thing. 19:07.560 --> 19:12.560 So if you want to, if you want to get further on this subject, there's a link to that RFC. 19:12.560 --> 19:21.560 And you have also what source.org, that is a website that was created by KPCYRD, he is an also Archenex, also an Archenex maintainer. 19:21.560 --> 19:34.560 And this website is very cool because you go to what source.org and then you tap for instance bash, and it will show you what sources every major list distribution are basing their bash package on. 19:34.560 --> 19:43.560 So you can see that, for instance, arch is basing as package against a custom made to our bowl and fedora does it against the get tag, etc. 19:43.560 --> 19:50.560 So that allows you to see what distribution are generally using for a package in terms of source. 19:50.560 --> 19:59.560 A few contacts information, and this is Archenex.org, if you want to talk to me, I have a website as well, I am on the Fedivers. 19:59.560 --> 20:05.560 And if you are going to get the slides, if you want to follow the links, and that's about it. 20:05.560 --> 20:16.560 Thank you. 20:16.560 --> 20:17.560 Thank you. 20:17.560 --> 20:18.560 Any questions for all? 20:18.560 --> 20:20.560 We've got a couple over here. 20:20.560 --> 20:21.560 Yes? 20:21.560 --> 20:25.560 Do you prefer to get tag over a get tag? 20:25.560 --> 20:28.560 I mean, yeah. 20:28.560 --> 20:34.560 So do we prefer a get tag instead of what do you mean get ash? 20:35.560 --> 20:37.560 We have both actually. 20:37.560 --> 20:46.560 Our package tooling is able to generate a a shot to 56 ash for VCS object, so we commit a tag etc. 20:46.560 --> 20:51.560 So we actually have both. 20:51.560 --> 20:52.560 Question? 20:52.560 --> 20:53.560 Yes? 20:53.560 --> 20:58.560 Would it help if the. 20:58.560 --> 21:02.560 If the step from the sources to off the gen or the. 21:02.560 --> 21:12.560 If we pay the title, would it help if the steps from going from the raw sources to those custom made to our bowl? 21:12.560 --> 21:17.560 Basically, so out of general, where reproducible or at least more transparency. 21:17.560 --> 21:18.560 Of course, that would help. 21:18.560 --> 21:31.560 This is one thing we are trying to do with upstream and and discussing with them is to give give us the information on what's actually done from going to this source to the custom made to our bowl. 21:31.560 --> 21:36.560 That allows us either to replicate it on our side. 21:36.560 --> 21:41.560 So we grab directly the transparent source or even better if it is reproducible. 21:41.560 --> 21:47.560 We could technically be based on the custom made source and with reproducible bills, it would be fine. 21:47.560 --> 21:48.560 Yeah. 21:48.560 --> 21:50.560 So that would definitely help us. 21:50.560 --> 21:51.560 Okay. 21:51.560 --> 21:53.560 One comment on that. 21:53.560 --> 21:55.560 For dead end. 21:55.560 --> 21:56.560 It's always. 21:56.560 --> 21:58.560 That's the rule. 21:58.560 --> 22:02.560 One to have source code in the prepared form of modification. 22:02.560 --> 22:06.560 Which means all the generated files are not prepared form. 22:06.560 --> 22:11.560 So we will always want to have the transparent form that have all our. 22:11.560 --> 22:15.560 Through the way the generator stuff and generated on our own again. 22:15.560 --> 22:16.560 Okay. 22:16.560 --> 22:18.560 I mean that that is obviously a good. 22:18.560 --> 22:19.560 Oh, I sure. 22:19.560 --> 22:21.560 So Debian does it differently. 22:21.560 --> 22:23.560 Basically what they do is that they grab the trouble. 22:23.560 --> 22:27.560 Remove the remove the custom stuff and redo the custom stuff itself. 22:27.560 --> 22:28.560 Basically. 22:28.560 --> 22:29.560 Yes. 22:29.560 --> 22:30.560 Sort of. 22:30.560 --> 22:31.560 Sort of. 22:31.560 --> 22:32.560 That's. 22:32.560 --> 22:33.560 The expectation. 22:33.560 --> 22:34.560 Okay. 22:34.560 --> 22:35.560 That's expectation. 22:35.560 --> 22:36.560 Okay. 22:36.560 --> 22:37.560 That's also a lot of work. 22:37.560 --> 22:39.560 I'm not sure if we are willing to do this. 22:39.560 --> 22:42.560 I mean Debian is a volunteer base as well. 22:42.560 --> 22:43.560 True. 22:44.560 --> 22:47.560 But yeah, of course that would work. 22:47.560 --> 22:52.560 But for this to work, we would also need to be like a policy wise and everyone doing that, 22:52.560 --> 22:55.560 which is also a bit difficult for us to do. 22:55.560 --> 22:57.560 But we have a lot of like more out of truth. 22:57.560 --> 23:00.560 We have a half while that doesn't generation. 23:00.560 --> 23:01.560 True. 23:01.560 --> 23:05.560 What could be good, or as well as just not using all of that would be even better. 23:05.560 --> 23:09.560 Other questions? 23:09.560 --> 23:10.560 Yes. 23:10.560 --> 23:12.560 There's some extreme cases through the work. 23:12.560 --> 23:14.560 What do language what we're going to work for? 23:14.560 --> 23:16.560 Where to generate a little smile. 23:16.560 --> 23:21.560 I don't think actually you need to see maybe even five and maybe some jazz stuff. 23:21.560 --> 23:23.560 Would you say it isn't always worth it? 23:23.560 --> 23:26.560 And sometimes absolutely doesn't want to cooperate. 23:26.560 --> 23:27.560 Because it works for them. 23:27.560 --> 23:29.560 That happens like. 23:29.560 --> 23:32.560 As so question is there. 23:32.560 --> 23:37.560 There are some upstream build system that are complex to work around. 23:37.560 --> 23:41.560 And sometimes upstream doesn't really want to cooperate with us. 23:41.560 --> 23:43.560 How do we handle that? 23:43.560 --> 23:52.560 Well, we actually decided to not rely on PiPiTarble for Python packages, mostly because of the same issue. 23:52.560 --> 23:57.560 And some upstream would actually tell you that they only expect PiPiTarble to be built. 23:57.560 --> 23:58.560 Yes. 23:58.560 --> 24:03.560 And if it's the PiPiTarble created by some data population. 24:04.560 --> 24:13.560 What if the steps to build this custom table are documented or inside the CI that we can load it? 24:13.560 --> 24:15.560 Of course, that's better. 24:15.560 --> 24:18.560 But it's also more work. 24:18.560 --> 24:23.560 Ideally, we should find a good middle ground that would work both for us and for them. 24:23.560 --> 24:25.560 So, yes, that would work. 24:25.560 --> 24:27.560 And if we have to do that, we will do it. 24:27.560 --> 24:32.560 This is not about reducing our number of packages because upstream don't want to cooperate. 24:32.560 --> 24:37.560 We want to keep our the same repository state that we have. 24:37.560 --> 24:44.560 But, yeah, that would be okay, but not the most desirable, basically. 24:44.560 --> 24:45.560 Yes? 24:45.560 --> 24:50.560 Is there a systematic way to check upstream change in the industry? 24:50.560 --> 24:52.560 We now have one. 24:52.560 --> 24:54.560 We did not have one before. 24:54.560 --> 24:55.560 Oh, yeah, repeat the question. 24:55.560 --> 24:59.560 How do we do if upstream recreated attack, basically? 24:59.560 --> 25:05.560 How can we identify that and eventually confront them on that front? 25:05.560 --> 25:09.560 Well, with what source.org, the website I said, 25:09.560 --> 25:15.560 WebSource.org is now registering every of the commit ash and commit ID of a tag. 25:15.560 --> 25:18.560 And if it changes, you can see that way. 25:18.560 --> 25:24.560 And it also gather a list of files from the previous tag and the new one. 25:24.560 --> 25:26.560 So, we can also compare the list. 25:26.560 --> 25:29.560 We're usually relying on what source.org. 25:29.560 --> 25:35.560 Or if our bill service, till have the old sources, then we can use that as well. 25:35.560 --> 25:36.560 Yes? 25:36.560 --> 25:42.560 So, that would be some view that you could actually use by two or all of the developers. 25:42.560 --> 25:46.560 And if they were to confirm, you shouldn't waste them. 25:46.560 --> 25:50.560 So, we do have a, yes. 25:51.560 --> 25:57.560 Our packages reproduce by third party people, or just by us, basically. 25:57.560 --> 25:58.560 Actually, both. 25:58.560 --> 26:03.560 So, we have a report of our reproducibility of packages. 26:03.560 --> 26:08.560 So, we have about 90% of our packages in the report that are reproducible. 26:08.560 --> 26:15.560 And this, this rebuilder is maintained by us, but on a different machine and the bill machine. 26:15.560 --> 26:17.560 Okay, I think that'll be the last question. 26:18.560 --> 26:24.560 And, and we also have like third party user that run their rebuilder instance and that also shows the result. 26:24.560 --> 26:28.560 So, you can compare our result with also third party result. 26:28.560 --> 26:30.560 And they are usually on par. 26:30.560 --> 26:31.560 Excellent. 26:31.560 --> 26:33.560 All right, I think that's it. 26:33.560 --> 26:34.560 Thank you. 26:35.560 --> 26:36.560 Thank you. 26:42.560 --> 26:43.560 Good.