WEBVTT 00:00.000 --> 00:15.280 I hope it's on, yeah. So, hello, I'm Fantyshek, I'm product owner for the couple of projects 00:15.280 --> 00:22.080 we'll be mentioning later on and I'll join by Christian from the TMT team and we'll be describing 00:22.080 --> 00:28.720 or discussing how we are trying to polish the way from the user down, down, down to the 00:28.720 --> 00:35.760 area to the real people. So, flow of code, that's the first topic then we would describe 00:35.760 --> 00:41.040 the packaging and testing experience projects, we work on, then the term of shift left 00:41.040 --> 00:49.040 ink and how we can, you can test, share the test and how you can collaborate. So, that's the 00:49.040 --> 00:57.680 plan. This is the space we are working on or around, we are used to fit around a central 00:57.680 --> 01:04.320 stream, but it works for other distributions in a similar way. So, we've really, after the development 01:04.320 --> 01:10.560 happens on GitHub, GitHub, GitHub or any other instances and there are plenty of ways how people 01:10.560 --> 01:17.360 test, there are GitHub actions, GitHub applications, GitHub runners and a lot of lot of stuff. 01:18.160 --> 01:24.080 On the Federas field, there is Gitforge, there are actually currently, there is a move from 01:24.080 --> 01:32.480 page, however it's pronounced, and to forge, you can, but not need to open a public rest to get 01:33.680 --> 01:39.680 get-out code there and there are generic checks and also you can define your own, 01:39.680 --> 01:45.360 there is also a possibility to run a test inside the RPM field, there are also post-built tests 01:45.360 --> 01:52.400 and there are also compost tests and to make it more messy on the center stream, it's different as 01:52.400 --> 01:59.600 well. So, we have to GitHub and here everything is supposed to happen before the magic 01:59.600 --> 02:06.400 rest is merged. You can also have like a separatist and RPM build tests and yet there are 02:06.400 --> 02:13.840 compost tests, but run completely differently than the previous one. And also when we are moving 02:13.840 --> 02:20.000 down the stream, we are no longer speaking about the projects, but packages, we don't longer speak 02:20.000 --> 02:29.120 about developers, but about maintainers, a lot of not-of-names. So, we have a group of related projects 02:29.120 --> 02:34.800 that recently got closer to get under one umbrella and also I'm currently like overviewing 02:34.800 --> 02:41.120 these projects and we call them packaging and testing experience, so the projects are related to 02:41.120 --> 02:48.640 like packaging, testing and also ideally this like work for the program. So, this is the mess, 02:48.720 --> 02:54.480 and this is the world how we are trying to treat this. So, yeah, we are all over the place 02:55.360 --> 03:02.240 in a some way integrated between various fields, so we'll try to decouple that a bit. So, 03:04.560 --> 03:10.880 one thing we are trying to do is to unify this. So, as I mentioned, there are a lot of 03:10.880 --> 03:16.000 differences between all these steps. We are trying to hide this from the user, ideally. 03:16.960 --> 03:23.120 Second thing, what we are trying to do is to make it more like a user-centered, so it's like a 03:23.120 --> 03:30.000 workflow from the user perspective, and also ideally change the manner of tasks as much as possible. 03:30.000 --> 03:37.600 So, it's like this change, change, and also ideally to not as exact as a 10 random projects, 03:37.600 --> 03:44.720 but ideally one unified interface, etc. Plenty of works still to do, but we are trying. 03:46.080 --> 03:55.120 Okay, so first, now I'll circle through the projects that are related to that just so, you know, 03:55.840 --> 04:03.840 first one is copper, it's an arp-embelled system and project management, and it allows anyone to 04:03.840 --> 04:11.200 provide an easy way, arp-em, yam repository, so if you know, PPA repository is from the 04:11.520 --> 04:21.280 world, that's the similar thing. Unlike if the cogey or blue, this is like, can be used for anyone 04:21.280 --> 04:28.080 to provide a repository. This one is log detective, log detective is a project is actually like a couple 04:28.080 --> 04:35.920 of pieces together, and the basic goal is like an arp-em block analysis, so you have an arp-em, 04:36.720 --> 04:42.960 your arp-embelled failed, and you have a huge log, and you want to get something meaning from 04:42.960 --> 04:46.800 top of it. Yeah, there are a lot of experience people, but there are also a lot of people that 04:46.800 --> 04:52.640 see this as like, oh, where is the thing that I should look, and also what should I do? So, 04:52.640 --> 05:00.000 there is a way how to put a log to that service, and it will run it through an AR analysis, 05:00.000 --> 05:07.840 and hopefully it will try to help you with that. Second piece is on the previous slide. 05:07.840 --> 05:17.520 The second piece is that we are actually collecting the like human defined data, so we are also 05:17.520 --> 05:22.560 collecting that you can provide your URL that you know, with the log you know, 05:22.560 --> 05:33.440 click on the snippets, so mark the relevant bits, and then describe them. So, we are 05:33.440 --> 05:41.120 trying to collect this database, so and providing the model out of it, so we can have a 05:41.120 --> 05:48.720 better input for this. And the last bit is actually the turning this as a service for you, 05:48.720 --> 05:54.720 so for sentencing merge requests. If there is an RPM built failure, you get this command, 05:54.720 --> 06:00.320 that tries to like describe the problem for you, and the team is currently working on 06:00.320 --> 06:05.120 getting these two federals, so close to upstream, so there's actually the place where more 06:05.120 --> 06:08.800 programs are happening. Hopefully, you can get far, far to the upstream. 06:10.880 --> 06:18.160 Third point is packet, which is an integrator or gitforge integration that tries to glue various pieces 06:18.240 --> 06:25.360 together. It can act as a CI, either for upstream and in a month or so for federals, 06:25.360 --> 06:32.640 git packages, but also it acts as a release automation, so if there is an upstream release, 06:34.640 --> 06:42.960 you can get automatic releases to federals and sentencing, and the federals also I mentioned these 06:42.960 --> 06:48.000 like gluing together and making a change in the activity, so if there is a disk, it would be 06:48.000 --> 06:53.040 as merged, you have automatic codej build, then automatic board here, I've determined the 06:53.040 --> 07:02.880 codej build succeed. So, as a CI, yeah, we are combining other two, so we have a GitHub application 07:02.880 --> 07:08.640 and GitLab integration where I have books, and with that you can have RPM builds, either for a 07:08.640 --> 07:14.960 copper or coggy, and then you can have testing farm tests, more about that later, either directly, 07:14.960 --> 07:22.880 or using that VM with the artifacts, you have installed, there is also pilot for the image builder 07:22.880 --> 07:29.520 integration that we talk about that in us, also, and also static analysis is available and we are 07:29.520 --> 07:38.720 working on the logitative path, and I'll pass my work to Chris Scher. Okay, so another topic of here 07:38.720 --> 07:44.720 is the TMT, it's a way to just manage all of your tests and your test environments, and the goal of 07:44.720 --> 07:51.600 it is to make all of the tests that run in the CI also be able to run locally, so you can define 07:51.600 --> 07:56.880 like more of the tests that you want to pass, and also where do you want to run them? I can 07:56.880 --> 08:02.160 know, destroy the container, what package to install and so on, and the main one of the other 08:02.160 --> 08:10.080 main goals is to share and reuse all of these kinds of test definitions, or share it like from 08:10.080 --> 08:17.280 upstream to downstream, or like all across like other packages to run as well, and the last part 08:17.280 --> 08:23.680 over here is testing farm, basically just run TMT tests inside the cloud, well, there's also like 08:24.640 --> 08:30.320 reading the TMT and like providing any kind of harder that we need, like GPU or whatever, 08:30.320 --> 08:35.440 and it also gives you like a more readable way of displaying the tests and how they felt. 08:37.760 --> 08:42.960 Okay, back. Yeah, and I will take a look at the term of shifting left. 08:44.240 --> 08:51.360 So what mean by that, that this is the flow of goal going down the stream, and we are trying to move 08:51.440 --> 09:00.800 as much testing and variation back to the left. So it's happening right, and the development 09:00.800 --> 09:08.400 is there, and it's not only like a move as a MV command, but it's ideally like putting it to the 09:08.400 --> 09:14.880 left, to upstream, but also verify on each step, because there can like people, there can be 09:14.880 --> 09:20.560 contributors and contributions happening all over the place, like on various of these steps, 09:20.560 --> 09:26.640 so we can't rely only on the upstream, and also like the environment can change slightly between these, 09:27.520 --> 09:35.360 but also like time happens, so yeah, slide different, also before something gets from upstream to 09:35.360 --> 09:43.040 better, for example. So what I mean by that, on the left, that the like usual workflow, that there is a 09:43.040 --> 09:48.720 pool request or a magic request happening in upstream, this gets merged, hopefully released, 09:48.720 --> 09:54.560 and in some time it gets to fedora, and in some time also it gets to cento stream, and if the 09:54.560 --> 10:00.400 user or the maintenance of the packages in a fedora and cento stream realize any issue, 10:00.960 --> 10:05.840 they need to go back to the upstream, file an issue or back to send a pool request, 10:05.920 --> 10:17.280 then there is a new PR that needs to be opened, and the circle goes back, then it gets to finally 10:17.280 --> 10:23.040 to upstream release, whether it's cento stream, it takes a lot of time. So what we are trying to 10:23.040 --> 10:31.520 promote is that let's get the feed back directly to the PR, so we are just making this a 10:31.680 --> 10:39.120 much, much, much quicker. So yes, so what we are there are two parts of that, one is the like 10:39.120 --> 10:44.720 verification, so learning, for example, build directly for upstream changes, and providing the 10:44.720 --> 10:51.200 environment for that, but also ideally, as you mentioned, by TNT, you can also provide the tests, 10:51.200 --> 10:57.840 and give them an upstream, and ideally with testing found, we are also providing the testing infrastructure, 10:57.840 --> 11:04.160 so it's not like, hey, upstream, give this testing, but also like trying to help them. 11:05.600 --> 11:12.080 Yeah, all that seems very nice, and we thought that like everyone will be happy about that, 11:12.080 --> 11:20.160 like we'll be like upstream and downstream, we'll be closer together, and yeah, but we had a lot of 11:20.160 --> 11:25.760 complaints for both of the sites, because yeah, not everyone is happy about that. So yeah, 11:25.840 --> 11:31.600 there is like a people from upstream complaining about that it's getting more work, they don't understand 11:31.600 --> 11:37.200 the distributions, and what don't want any file in my gate, from the other perspective, from the 11:37.200 --> 11:43.520 maintainers, we are getting like, oh, we are also maintaining those, and don't want to care about the 11:43.520 --> 11:50.240 view upstream, we are those that understand the RPM stuff, and too many upstreams to handle, and 11:51.200 --> 11:55.520 we actually don't trust it in some time, then. So let's take a look at them, one, one. 11:56.560 --> 12:05.200 So, more work for us, it's actually not so true, because the issue, if you see the arrows, 12:05.200 --> 12:12.960 will happen, or you will get it nevertheless, but with the shift shifting left perspective, 12:12.960 --> 12:18.560 you see just right when the development is happening, so when there is a new change, 12:18.560 --> 12:25.120 you see that there is a problem with it, you can fix it, no context switching, you just have it 12:25.120 --> 12:34.960 directly there. Also, we can help with the infrastructure, so it's not so tough for you. Second, 12:34.960 --> 12:40.400 complain is that people don't understand it, and that's correct, and ideally there should be a 12:40.400 --> 12:46.960 relation between upstream developers and downstream maintainers, trying to help each other in some 12:46.960 --> 12:53.920 collaboration, and also with a few projects, we realize that there is a way how to, like, 12:55.120 --> 13:02.320 include downstream maintainers to certain situation when they help us need it, so for example, 13:02.320 --> 13:08.240 automatically if there is an RPM build that failed, we can just create the automatically, 13:08.240 --> 13:13.360 comment that he, he, these people, that are from up downstream, please take a look at that, 13:14.080 --> 13:17.920 so they are notified only when that something irrelevant is happening. 13:21.040 --> 13:27.680 Yeah, I don't, there are projects that don't like any digital specific file, anything, 13:27.680 --> 13:34.800 maybe some tests, so we have, we provide a couple of things that can help here, 13:34.800 --> 13:39.840 we are kind of flexible and we're trying to be kind of flexible, so we can reference, for example, 13:40.800 --> 13:46.720 we have also very, how to, like, load a lot of stuff, it's downstream, for example, spec files, 13:46.720 --> 13:53.760 but there are absolutely, don't like that, streams, yeah, understandable, so we have also poor 13:53.760 --> 14:00.080 workflows, for example, for the release automation, we also have a very, how to, how to avoid this, 14:00.080 --> 14:07.120 and we just pull the, pull the stuff from upstream, trying to glue it, but yeah, we don't have that 14:08.000 --> 14:13.840 history, from the other point of view, if you don't trust the upstream and you are maintainer, 14:13.840 --> 14:21.680 you are just providing a list of patches and put trying to maintain it and you are just long-term 14:21.680 --> 14:32.720 getting a lot of work for yourself and yeah, so ideally, you are just losing time and trying to 14:32.720 --> 14:40.320 maintain something on top of upstream and yeah, a lot of work for you, same, if you don't 14:42.160 --> 14:47.280 to do the real stuff, you're actually doing the same thing, you are just doing it elsewhere, so 14:47.280 --> 14:53.520 ideally, you are helping to fix these problems right on upstream part, and if there are too many 14:53.520 --> 14:59.920 upstreams, I've mentioned it already, the ideal case is if the upstream developers and downstream 15:00.880 --> 15:07.760 are somehow in contact or trying to help each other communicating, I think there is a benefit 15:07.760 --> 15:15.360 for both sides, so and long-term maybe the upstream developer will start understanding the basics, 15:15.360 --> 15:22.560 so can actually start commenting, to downstream stuff, so hopefully long-term it's definitely for 15:22.560 --> 15:29.680 it. So let's look at this kind of shifting left in practice with 15:29.680 --> 15:36.000 how we do some test sharing, so there are two ways of sharing the test, one of them is like sharing 15:36.000 --> 15:41.840 between upstream and downstream, so over here you just care about like do the installation, don't 15:41.840 --> 15:48.000 care about like how the stuff you get to there, back it or whatever other CI will take care of that, 15:48.800 --> 15:53.280 and it's kind of nice because like it can bypass some of the limitations that other 15:53.280 --> 15:58.480 disk has, for example, in Fedora you cannot run tests that require internet over here you can, 15:59.760 --> 16:07.040 and it keeps things a bit cleaner because you can also use some files from from the upstream 16:07.600 --> 16:15.520 get repositories and their sources and so on. Another way that the test are shared is like across 16:15.520 --> 16:21.040 different packages, the simplest thing is just run the same tests for all the packages, 16:21.200 --> 16:30.000 RPM inspect, installability, and so on. Another one that some people have used is like some kind 16:30.000 --> 16:37.680 of reverse dependency tests, let's say like one project has a built failure, one project has like 16:37.680 --> 16:43.200 some tests defined, and then another project depends on the other one, then you can just run 16:43.200 --> 16:49.040 backwards like the test from other projects and see if any of your changes like affect the other ones. 16:51.120 --> 16:57.520 We can take an example with a concrete project like psychic build core, it's like one of 16:57.520 --> 17:02.960 one project that has adopted this kind of workflow, and all of the definitions of the tests are 17:02.960 --> 17:08.160 inside the upstream repo, and you can just see like just like some simple tests that 17:08.960 --> 17:16.560 Python important see how it goes. They also implemented some tests that are run only in the upstream, 17:16.640 --> 17:22.800 so for example like their Python, and this is useful for checking that it works with different 17:22.800 --> 17:29.280 the GCC that is currently in draw height and will not appear and will catch some errors much quicker. 17:31.360 --> 17:38.080 And with testing farm and packet all of those tests are run inside the GitHub actions, 17:39.040 --> 17:45.200 and you get like some nice results, yes, smoke test, there is a version sure. 17:46.880 --> 17:52.320 And they also have like all of the tests from the upstream does all of the Python so on. 17:53.680 --> 17:59.600 On the upstream side same story, you also have like fed RCI or packet or whatever 17:59.600 --> 18:06.240 run equal of this test, and then you can check the same kind of interface, you get testing farm 18:06.240 --> 18:11.200 showing all of the downstream tests that were defined in upstream, around over here with a bit of 18:11.200 --> 18:19.520 different context. Anyway, so get like all of the tests that are running for all the packages, 18:19.520 --> 18:27.760 this is the instability one, and so on. Now, how can you get involved? Like we really, 18:27.760 --> 18:33.440 we really like for other disorders to also like get this kind of like workflow of like shifting 18:33.440 --> 18:39.360 left and like contributing more towards upstream so that we as fed RCI can also benefit and 18:39.360 --> 18:46.960 maybe also you. So what you can do is like just like try to contribute more to upstream, 18:46.960 --> 18:53.280 and maybe also like have this kind of TMT test. So in principle, this should work for all 18:53.280 --> 18:57.280 distributions. If it doesn't work, please come to us and like let's try to make it work. 18:59.600 --> 19:07.360 The files are like relatively simple. We'll probably have like some better documentation later on 19:07.360 --> 19:12.480 of like the minimum setup, but it's basically just defining what package you want to install 19:12.480 --> 19:18.640 and then the test over here, like at the Hello World and so on. And you can also run them locally. 19:18.640 --> 19:27.040 So just do a command like TMT run and it will do the same thing on your locomotion. 19:28.560 --> 19:33.840 And try to like find out like what is the basic kind of functionality the package does. 19:34.560 --> 19:41.200 So maybe just a Hello version over there or whatever. And that would help like 19:41.200 --> 19:46.960 unify all of the tests between distributions as well and share them all across us. 19:48.400 --> 19:54.000 And kind of so like think of like what kind of like limitations you have in the CI of like 19:54.000 --> 20:01.840 GitHub not having many runners or so on. TMT and test in front we can have like multiple machines 20:01.920 --> 20:06.400 that you can run tests and like you can have a server and a client and you can do some 20:06.400 --> 20:07.600 color operations as well. 20:09.920 --> 20:14.240 Otherwise you can get the full is like contribute to like some of the shared tests. Like in 20:14.240 --> 20:19.440 Federation CI we have this kind of shared RPM inspect test, installability and so on. 20:20.000 --> 20:25.600 Sometimes they fail yes and we could get like some more people like looking into them and like see 20:25.600 --> 20:38.560 how you can contribute. For us you can find us like other Federation CI matrix from and that's 20:38.560 --> 20:43.760 about it. Yeah also in the package can also find all of us. 20:56.560 --> 21:08.800 So there are any questions. So I can get a nice question before but as we have a lot of 21:08.800 --> 21:16.240 the tests in both places they're getting understanding the outputs as a user and it's a making 21:16.240 --> 21:22.880 a developer user is about how many things can you take in your team. 21:23.440 --> 21:29.440 Yeah so the question was that if you came out of testing and maybe increasing that how to 21:29.440 --> 21:37.200 understand the output yeah that's a tough one definitely like one is the representation yeah the 21:37.200 --> 21:44.160 testing from visualization needs some love and we are discussing that so like the formal and 21:44.160 --> 21:52.480 the second thing is that like it would be nice to get it to the right place so maybe like 21:52.480 --> 22:03.440 in a good time you get this but yeah I can definitely definitely might help as a little 22:03.440 --> 22:10.480 log detective there are some initiatives like doing a very similar thing from that and 22:10.480 --> 22:17.280 Chris you might have more ideas. From the TMT side I think there is one way you can kind of improve 22:17.440 --> 22:24.000 this is to manage your tests. So not having for example love for the pilots don't run all of 22:24.000 --> 22:32.720 the pilots at one like just run individual individual tests and like have more concrete what has 22:32.720 --> 22:40.400 felt then you can try running locally and then with only that test and see exactly what is 22:40.480 --> 22:47.760 the minimum environment to reproduce also there's another thing could do is like TMT has this kind 22:47.760 --> 22:54.800 of login operation so you did the TMT run and then login and whenever it felt it gives you a shell 22:54.800 --> 23:02.400 script a shell environment and then can try debugging inside there. Yeah maybe to go to it 23:02.400 --> 23:08.000 a TMT has this like higher concept it can't it's a different thing then for example 23:08.000 --> 23:12.640 fighters and these frameworks it's like a more high level where you can document also the group 23:12.640 --> 23:19.200 of test disciplines but the second thing TMT also allows to report the test somewhere the 23:19.200 --> 23:25.200 issue is that we have so many reporting places and not every man uses that so that's maybe 23:25.760 --> 23:31.040 what I would like to look at to maybe get rid of all the duplication also on the reporting side 23:31.040 --> 23:38.880 because there are so many places where you can put this if anyone has a good idea better idea please 23:38.880 --> 23:49.840 very sure we'll be happy to improve you are a specific one last question or any complaints 23:49.840 --> 23:55.800 rather than not good? Thank you.