WEBVTT 00:00.000 --> 00:14.000 Hello. My name is Paul Floyd. I'm here to talk to you about, as I said, a vaguined on 00:14.000 --> 00:21.400 dragonfly net and open BSD. I'm a valgrain developer. I've worked mainly on free BSD, so I'm 00:21.400 --> 00:28.320 not going to talk about that much today. My talk is going to be in two parts. First of all, 00:28.320 --> 00:33.520 a brief overview of our grind. I only have 30 minutes, not 20 hours, so I can't 00:33.520 --> 00:41.280 go to details. Second part will be on a vibramed on the BSDs. Our valgrain, it's mainly 00:41.280 --> 00:49.920 new loan as a total for detecting memory faults for developers. It also has tools for 00:49.920 --> 01:00.240 thread hazard detection and profiling. So when valgrain runs, what it's doing is taking a bunch 01:00.240 --> 01:08.640 of instructions from what we call the guest. That's the test execution. It converts that into 01:08.640 --> 01:16.240 intermediate representation, a bit like a compiler. It will instrument that to some optimization, 01:17.200 --> 01:23.760 write it back to memory and execute it. This is the core of our grind. This is the schedule 01:23.760 --> 01:36.400 that's doing this all the time. It's also a website. In this scheduling loop, there's also 01:36.400 --> 01:41.600 doing other things. It has to handle how the end of these basic block occurs, has to handle 01:41.680 --> 01:48.240 things like errors and system calls. That's really the core of our grind. Sounds easy. 01:49.840 --> 01:59.680 Just one process. So the guest and host, both in memory, let's get very separate. But not 01:59.680 --> 02:06.720 like GDB or LDB where you have P trace and two processes. A big thing here for developers is the 02:06.720 --> 02:13.760 tool. It's always statically linked. We don't link with LBC. I sometimes say it's C minus minus, 02:13.760 --> 02:20.800 there's no LBC. We do that because we don't want to have any conflict between the tool using 02:20.800 --> 02:29.440 LBC and the guest using LBC. If you do a get-check out of our grind, you'll get about 840,000 02:29.440 --> 02:37.040 lines of code. It's fairly big. About two thirds of that is the code and one third is test cases. 02:38.800 --> 02:46.080 The big blue part here, Vx, is the synthetic CPU. And that covers all the platform that the 02:46.080 --> 02:53.600 grind will run on. It's pretty big. I'm only going to talk about AMD 64 because that's 02:53.680 --> 03:03.920 any platform that the BST run on. Free BST also runs on X86 and ARM 64. Good news here is that for 03:03.920 --> 03:12.960 operating systems, the Vx code is very platform independent. I've worked on Free BST for five 03:12.960 --> 03:21.920 or six years. I've made one change to Vx, but that was for a kind of memory fault, which on Linux it 03:22.000 --> 03:27.440 pleases a segmentation fault on the BST's SIG bus. That's the only thing. 03:29.040 --> 03:33.840 Bad news for us and in validering developers, we don't have enough people for all the CPU 03:33.840 --> 03:43.440 features so we struggle a lot to do everything we. That's our problem. The big red part here is 03:43.440 --> 03:50.800 core grind, so this is all the core services that Vagrin has to be provide. Like I said, no 03:50.800 --> 03:55.920 lib C, so we have our lib C replacement there. Then there's all the glue that we have to provide 03:55.920 --> 04:05.120 to make Vagrin look as though it's a real CPU and operating system. Next thing here, if you're 04:05.120 --> 04:10.800 not familiar with it is the GDB server, which allows you to connect GDB to an executable 04:10.800 --> 04:17.600 in Vagrin and debug it. It's all part of the core band. This part is very, very platform dependent 04:19.680 --> 04:26.400 extremely. Just reference about 15,000 lines of code for Free BST in work. 04:28.960 --> 04:34.640 Not huge, but a very big. The small little bits of the pie chart, most of those are the tools 04:35.600 --> 04:41.680 and as you can see the tools are pretty small by comparison. Memcheck, the most well known tool, 04:41.680 --> 04:49.680 about 3% of the source code. I'm not going to talk about much of the details about Vagrin running, 04:49.680 --> 04:54.320 just a couple of things that are important. The one of the most important things is to start the 04:54.320 --> 05:00.320 guest at Ctable. All this is happening in Vagrin. Vagrin is doing the loading and it'll run 05:00.400 --> 05:07.520 LD.SO from within Vagrin. Here's an extract of a small hello world XC. That's the 05:07.520 --> 05:14.400 object output on Free BST. You can see the load, that's the pieces of the file that need to be 05:14.400 --> 05:21.360 loaded to memory to execute it. If you can't read the out information and look at it runs, 05:21.360 --> 05:26.800 it's just going to tune it. It's good to finish. Similar to that, there's also the symbol table. 05:27.440 --> 05:34.240 We need to read the out symbol table. If we can't read the out symbol table, then we can't do 05:34.240 --> 05:39.040 things like replace malach or wrap the pie thread functions and lots of things work. 05:41.680 --> 05:48.720 When libraries are being loaded, Vagrin decords all this. In theory, Vagrin does has a perfect memory 05:48.720 --> 05:56.160 map of all everything's in physical memory. It doesn't work correctly, then we have problems 05:57.200 --> 06:03.680 likely to be a crash or just mad results. The last one here is Dwarf the debug information. 06:05.200 --> 06:12.560 If we can't read that, no source information in every message is and it becomes almost useless. 06:15.360 --> 06:24.480 So two difficult things signals on threads. For signals, when there's a signal handler, 06:24.800 --> 06:31.360 we have to make sure that the signal handler function doesn't run on its own on C2PU. We have to 06:31.360 --> 06:36.720 hijack it and make sure it runs in Vagrin. Similarly for the secret term, that'll go back into Vagrin. 06:38.400 --> 06:43.520 It's the same thing for P threads. We have to hijack the P thread to make sure it runs in Vagrin. 06:46.400 --> 06:51.600 For signals, normally when Vagrin does run, it's masking all signals. 06:52.560 --> 06:57.520 If it fall controls, when signal gets delivered. The exception to that is when there's a 06:59.760 --> 07:05.520 blocking system called, for the guest, then we have to restore the guest signal mask. Do this 07:05.520 --> 07:10.720 to some call, return to the system call and block that mask all signals again. There's lots of 07:10.720 --> 07:16.480 juggling of system calls that are going on. It's very, very hard to get all this absolute 07:16.480 --> 07:24.080 endemic direct. For the BSTs, I've seen quite a few problems with this. I've talked about 07:24.080 --> 07:35.680 system calls. This is the biggest part of any platform for an OS. So we have to wrap every single 07:35.680 --> 07:42.400 system call. So those of you who are not familiar with this system calls, 3BSD has 500 or so 07:43.040 --> 07:50.080 typically run in the 500 reason. So it's a lot of code. Good news here. Most of this is just 07:50.080 --> 08:01.200 boil code. This example from FBSD is the FF startup system call. You can see the print macro, 08:01.200 --> 08:08.400 this is used for logging. Vagrin has many, many logging and tracing facilities. One of 08:08.400 --> 08:13.920 these system calls. There are a few macros that we use for checking memory reads and writes. 08:14.560 --> 08:19.280 There's a macro there for checking the arguments to be system call. And there are a couple 08:19.280 --> 08:25.520 other things we do. There's tracking and checking file descriptors. And we have to do things like 08:25.520 --> 08:31.680 flag when there's a blocking system call. Most of the time it's easy, but somewhat quite difficult. 08:31.680 --> 08:40.320 So any system call involving threads, processes and querying CPU. That can be a lot more difficult. 08:44.400 --> 08:48.720 That was the code to our tests. So this is my main measure of call to you for Vagrin, 08:48.720 --> 08:54.640 the regression tests. Built into the source we have between six and eight and 08:55.120 --> 09:01.600 depending on the CPU, tests, there's a not OS specific. Okay. 09:03.040 --> 09:08.400 We'll make some free BSD. They have about 100 tests each. And I put in red here, right now, 09:08.400 --> 09:14.640 drag and fly open and net BSD is zero tests with just for them. But in any feature that they have 09:14.640 --> 09:21.680 on just for that platform is untested. So not good for the copy. Okay, a few more things we have 09:22.560 --> 09:29.440 some bigger tests, one based on the GNU scientific library. And Linux only, shame we don't have 09:29.440 --> 09:36.400 it in free BSD, LTP, the Linux test project. The last one there, we have a small performance test, 09:37.200 --> 09:39.840 which we don't run very often because we know that flag went very slow. 09:40.480 --> 09:52.480 Vagrin developers and Vagrin world, we're not very many, six people all working through part-time 09:52.480 --> 09:59.280 on it, mostly it's red hat. So there's a team working on GDB, binary tools like 10:01.760 --> 10:06.960 dwarf tools, elf tools and set that and Vagrin. So it's just spending some of that time on Vagrin. 10:07.520 --> 10:12.000 At IBM, a couple of guys working on system 390 and PowerPC. 10:12.880 --> 10:18.720 Men as me, I'm working in my spare time, mostly on the core green part and free BSD. 10:21.760 --> 10:30.720 So for platforms that are covered by our, the Vagrin world, most of our stuff is on sourceware, 10:31.680 --> 10:37.920 website, git and modern CI, our modern CI uses billbot. 10:40.000 --> 10:44.800 Vagrin has been around for 23 years, so we've got a lot of history. For historical reasons, 10:44.800 --> 10:50.960 our bugula is on KDE and sourceforge, if you remember that, it still has our mailing list. 10:54.720 --> 10:59.520 Vagrin developers, we can access the GCC server farm and that's good for us because that gives us 10:59.600 --> 11:05.200 a variety of operating systems, CPUs, and there are a few BSD machines on the way. 11:06.880 --> 11:13.200 The last day there's our old Infra, for CI, that's a script you can put in, Chrontab, 11:14.480 --> 11:19.920 which checks if there's been a git commit that day, checks out, build and sends an email to one of 11:20.000 --> 11:30.800 them, mailing lists. BSD's net, all great I'm talking about, for Vagrin fly next in the 11:30.800 --> 11:37.600 corner BSD, the source code I've looked at it was based on the free BSD fork, but all fairly old, 11:37.600 --> 11:48.080 all from about five or six years ago. Vagrin fly, screen shot of a guy called Dan is GitHub site, 11:49.040 --> 11:57.040 he did some work in 2021, hasn't been continue to don't think unfortunately. 11:58.240 --> 12:03.200 When I looked at that source code, it looked very similar to the state of the free BSD source code 12:03.200 --> 12:11.920 in Vagrin, when I started working on it, five or six years ago. Because Dragonfly is the closest 12:12.000 --> 12:17.440 historically to free BSD, it should be the easiest to fix and to take the things that I've done on 12:17.440 --> 12:27.440 free BSD and get to improve. Net BSD, so very hard to get to the source code, the wiki 12:27.440 --> 12:33.600 refers to a defanct billionus website. I couldn't find the subversion thing anyway, package source 12:33.600 --> 12:39.680 only refers to Linux, but what I'm going to talk about is I was contacted by some final year 12:39.680 --> 12:44.400 students at the Western Washington University who were working on a family project to get net 12:44.400 --> 12:53.600 BSD Vagrin working. Difficult for students for project. They did a good job, but unfortunately 12:54.320 --> 13:00.160 they were a few very serious small problems causing those things to still fail. It's a shame 13:00.160 --> 13:06.960 they didn't contact me at the end of their project to give it more help. I'm not going to talk about 13:06.960 --> 13:14.240 all the bugs and fixes I've done, just a couple, so here's one hand fixed yet. Net BSD, this is 13:14.240 --> 13:20.480 for a PIE executable, position independent executable. You can see the bottom line there, 13:20.480 --> 13:27.760 Vagrin is complaining, it's trying to execute an instruction in readwrite anonymous memory. 13:28.800 --> 13:33.600 That's totally wrong, that should be read execute and file mapped. So I don't know what the 13:33.680 --> 13:43.520 cause of this is. Non-PIE executables run a K and PIE does work on 360 and Linux. It's not a fundamental 13:43.520 --> 13:51.920 problem. Just one example of a little bag of effects. In the code there's lots and lots of 13:51.920 --> 13:58.880 Defines, it's very platform dependent, this is operating system OS and CPU Defines. This function 13:58.880 --> 14:05.120 recording the start up directory was missing the net BSD Define. The function had an old pointer 14:05.120 --> 14:10.960 for the start up directory, which meant when things like log files or results files would 14:10.960 --> 14:16.800 open at the end of execution, it would fail. Lots of things are failing because this one's small 14:16.800 --> 14:24.640 error. Open BSD, this is the element in the room, this is the pin Cisco teacher. This was 14:24.640 --> 14:34.880 adding in 7.5. This is the feature that is intended to limit system calls to LD.SO and Lipsy. 14:36.080 --> 14:41.360 This is bad news for Vagrin because we need to make direct system calls. I've been talking about 14:41.360 --> 14:47.200 system calls but a lot already. One solution would have been to do like Pearl and to use a dispatcher. 14:47.440 --> 14:56.080 That's a bit of a non-starter. Not all Lipsy functions have rapid system calls so not 14:56.080 --> 15:00.480 having is there. Lots of the biggest problem is you have to get Lipsy loaded and on free 15:00.480 --> 15:06.880 BSD there are about 1,800 system calls even for Lipsy gets loaded so not possible. 15:09.920 --> 15:15.760 The only thing I can see for Open BSD would be to implement a pin Cisco's table in Vagrin. 15:16.640 --> 15:22.800 This is well beyond my abilities with Open BSD. Here's an example from a blog site I saw 15:23.920 --> 15:32.160 using just two system calls. All the testing I've done on Open BSD was with 7.4. That's before 15:32.160 --> 15:39.600 pin Cisco's was fully implemented. Now I'm getting out of the important things. 15:40.400 --> 15:50.960 Pay attention. For comparison, FBSD and Illumos. So this is showing which is the current version 15:51.920 --> 15:56.800 before I start at least. Whether it's in upstream or not, whether it's up to tree. 15:58.560 --> 16:02.240 Whether you have a buying package or not and the quality of the regression test. 16:02.960 --> 16:08.640 So pre-based in Illumos, up source, package is available and virtually all the regression tests work. 16:10.560 --> 16:15.680 Dragonfly, I saw there was an attempt to get the binary package working but that 16:15.680 --> 16:19.920 fizzled out for some reason. But two thirds of the test cases were passing. 16:20.640 --> 16:26.320 So what I've done is I've rebased the code fixed a few things. I'm getting about 78% of the 16:26.320 --> 16:32.880 test cases passing. Net BSD, I guess at the students they had a few small problems causing 16:32.880 --> 16:38.880 lots of errors. I fixed those rebased and now I'm getting about two thirds of the test cases passing. 16:38.880 --> 16:46.640 That's not brilliant, but it's good. Open BSD, a lot of small problems there. I haven't rebased 16:46.640 --> 16:53.200 that code. I don't think anyone had ever run the regression tests because some of the components 16:53.200 --> 16:58.960 haven't been ported to open BSD. I fixed those. I was only getting about 20% passes. I fixed a few 16:58.960 --> 17:06.160 things. I'm now getting about 40%. That's the, if one thing you remember here is the two thirds 17:06.160 --> 17:12.960 of a net BSD test passing. Those figures are a bit on the pessimistic side. The regression tests 17:14.000 --> 17:19.840 have lots and lots of comparisons with core stacks which require a lot of filtering to get, say, 17:19.840 --> 17:24.320 a few BSD calls back to look like a Linux core stack. And I haven't done that work for these platforms. 17:24.320 --> 17:34.720 So real picture is a bit better. So for all the work I've done trying to fix a few things 17:34.720 --> 17:40.240 and update things. I've created a GitHub repo. There's a 4.726 branch for each of those. 17:42.720 --> 17:47.840 Another big question, where next? There's too much there for me to do all by myself to get them all 17:47.840 --> 17:54.080 to be working fully. I need help from these communities. I need people to get in touch with me 17:54.880 --> 18:03.120 to carry on the work. So I can help, I can advise, for like I said, I already spend a lot of time on 18:03.200 --> 18:08.480 general vibrant and free BSD in the Linux. And I think free more. I also do some work on macOS as well. 18:10.960 --> 18:14.880 It's just too much. So I need help. If you want to have vibrant working well on these platforms, 18:15.440 --> 18:23.920 get into to me. We're old school. If you want to get in touch, mailing lists and IRC. 18:26.880 --> 18:31.120 We also have some very open source people as well who don't like, particularly things like 18:31.760 --> 18:40.640 Discord and things like that. Okay, a few more resources. I gave a talk four years ago during COVID. 18:42.640 --> 18:52.240 It's always a very general thing. The last resource here at YouTube. There's a talk given by 18:52.960 --> 18:58.080 good butcher's name, Mazda, Ubashi, at the European SDCon for 12 years ago. 18:59.440 --> 19:05.200 I could recommend that. It was the one I would talk. So it's more technical and goes into a lot more detail. 19:06.800 --> 19:16.720 And lastly, our website and the readmees and our gigs, so let's go. Thank you for listening. 19:22.240 --> 19:24.400 Thank you.