WEBVTT 00:00.000 --> 00:22.520 Well, welcome to my talk. I'll go over what happened in the past year on in terms of mainline 00:22.520 --> 00:28.840 kernel and he would support in Raptor. Also, may so. So first of all, who am I? I'm Nicola 00:28.840 --> 00:32.360 of Raptor O'Reilly. I'm from Switzerland, which explains why the first name is 00:32.360 --> 00:41.920 France, the French way. The second name is Italian and the accent is German. Yeah, I got 00:41.920 --> 00:49.480 started working on rock-tip stuff in 2021. First thing I upstream was the year 00:49.480 --> 00:57.160 object by the STDM Audio Driver. And I joined Colabora basically a year ago as a kernel 00:57.160 --> 01:04.440 engineer and I've made several contributions to rock-tip in subsystems in Linux in that 01:04.440 --> 01:10.280 year. That's good luck among this probably, undercounting, but it's the basic of quick 01:10.280 --> 01:16.920 little pup. However, this talk is not just about like what Colabora did, but the community 01:16.920 --> 01:23.080 has a whole. So there were a lot of outside contributions as well. First of all, 01:23.160 --> 01:29.960 I rock-tip. The boards are very affordable and widely available. So before the great 01:29.960 --> 01:38.360 time for the kit, you could get the flagship SLC boards for like 70 to 180 euros depending 01:38.360 --> 01:49.080 on the amount of RAM you wanted. That's 4GB to 32GB. Many of them have their origins in the TV 01:49.080 --> 01:53.960 boxes and stuff, which is why they have very good multimedia capabilities for like their 01:53.960 --> 02:01.560 price range. They actually have a lot of their own hardware decoder encoder designs, which is 02:01.560 --> 02:08.760 how they were like the first company that I'm aware of that had H264 10-bit hardware decoding, 02:08.760 --> 02:16.280 which is a very obscure profile or at least was until HGAR had to see and HVC licensing became a mess. 02:17.240 --> 02:23.640 Also, the TORMs are extremely, easily available. Sometimes, officially from rock-tip 02:23.640 --> 02:28.600 themselves, other times you can find them floating around on the internet. A lot of times, 02:28.600 --> 02:33.240 like the SBC vendors have just a copy of them up there. I don't know how sanctioned 02:33.240 --> 02:38.040 that is, but even if you get them from rock-tip officially, you will like see a big water 02:38.040 --> 02:44.920 mark or rock-tip confidential on the TORMs. So evidently that is not to be taken too seriously. 02:47.160 --> 02:55.480 And yeah, the vendor code is also like, you'll do notice that their engineers are like 02:55.480 --> 03:00.120 frequent contributors to upstream not just fundamentally, but also how the drivers are written. 03:00.680 --> 03:07.880 In that it does look like Linux kernel code. They're still like that, the vendor drivers. They're 03:07.880 --> 03:13.960 relatable. I'm 50 hours into a six-year-old work week and I need to shift this and I don't have 03:13.960 --> 03:20.760 time to refact the entire subsystem kind of bad. It's not the whole let's write tens of thousands 03:20.760 --> 03:28.760 of lines of whatever kind of bad. So it's good to go over the naming of the sheet so quick because 03:28.760 --> 03:33.560 that makes you kind of a rare of where each chip in stack is, but it was here in the name. 03:34.280 --> 03:38.280 This is just one naming scheme to have other chips. We don't have time to go over the other chips. 03:39.240 --> 03:45.800 First, you have the ORK3 prefix and then you have one digits. Did I have labeled X here and then you have two 03:45.800 --> 03:55.400 digits falling that. The X is kind of the generation. You will notice if you pay attention, there is no fourth generation. 03:56.120 --> 04:00.520 Now, I don't get a guess. I went on Wikipedia. I looked up Chinese numerology and saw that 04:01.480 --> 04:07.880 for is an unlucky number. That kind of plays into the next point which is that generation 04:07.880 --> 04:15.080 here is mostly a marketing term. They will happily re-work the entire part of designs in chips 04:15.080 --> 04:20.040 within the same generation. It's not that just because it's in the same generation, you're going 04:20.040 --> 04:24.920 to have this driver work on this other thing in this generation. It's not how it works. 04:25.800 --> 04:32.360 What follows is kind of the performance tasks of the chip in that generation, the higher the 04:32.360 --> 04:37.800 better and you will see a lot of repeats, digits, eights there. It's also because eight is considered a 04:37.800 --> 04:44.840 lucky number, I assume. Sometimes you have like suffixes, that is one letter, but those you know 04:44.840 --> 04:49.960 can vary widely and usually, just assume it's a different chip. 04:49.960 --> 04:58.200 So some examples, the current flag shape is the RK3588. It's a four performance for 04:59.800 --> 05:05.880 efficiency core design. There's a Mali Valholt CSF GPU which is the first kind of 05:06.360 --> 05:11.880 GPU, Mali GPU design that was explicitly designed around kind of the multiple 05:11.880 --> 05:18.760 GPUs of work and stuff like that. They also didn't scheme about on the memory control or 05:18.760 --> 05:27.000 you get quad channels of up to LPDDR5 and it performs the get is about 2022 entry level Intel Chromebook 05:27.000 --> 05:36.200 which is quite good for an SPC. Then one SOC that might be a bit confusing, RK3588S, it's a cut 05:36.280 --> 05:43.400 down version. I don't think it's a bit thing, I think it's legitimately just different silicon, 05:43.400 --> 05:48.120 but stages have removed a whole bunch of IO stuff, however the CPU cores and GPU is still the same. 05:49.160 --> 05:57.320 When we will talk about a lot today, it's the RK3586. That's kind of the cheaper entry points 05:57.320 --> 06:05.400 to the kind of the stuff that can be found on the flagship and then we have the RK3589 which 06:05.480 --> 06:10.520 was the previous generation flagship and has great mainline support because it was in a Chromebook 06:10.520 --> 06:18.200 and at a time Google kind of said if you make a Chromebook with a chip in it, you have to upstream 06:18.200 --> 06:22.520 support for the chip so that's how a lot of the upstream for the chip happened. 06:24.840 --> 06:34.280 Then we don't have time to talk about those. So when you mainline stuff, one of the fun things is 06:34.360 --> 06:41.880 that generic UEFI ARM system already kind of images to standard work like fedor IoT, 06:42.520 --> 06:52.040 but the good benchmark is also, can you just treat like a PC? Can you use a generic R64 UEFI installer 06:52.040 --> 06:58.440 on a USB thumb drive to lock it into a board, install that distribution to local storage and 06:58.600 --> 07:09.320 boot from that local storage. So that's what I've done here. It's a RK3586 board and I've got an 07:09.320 --> 07:19.640 NME drive clocking below and I've got open-source tumbleweeds on the USB thumb drive, HM1 and Torkey board 07:19.880 --> 07:27.560 and so forth. So I'm trying to get open-source tumbleweeds installer and first kind of hurdle. 07:28.440 --> 07:34.760 It initializes a video output clearly but it displays the installer only as text on serial. 07:35.320 --> 07:40.440 Now that could have been my fault because I skipped the grubbed prompt by entering it on a 07:40.440 --> 07:47.720 hitting enter on serial and that's because usually you don't see that because we don't have video 07:47.800 --> 07:55.400 output in UEFI yet. But once you do that then maybe it will work in a way. It stumbles through 07:55.400 --> 08:02.520 the finish line. It shows a fatal error well installing that is not actually fatal. It works as fine. 08:03.080 --> 08:10.200 It boots from NME after installation but into the terrible unaccelerated X11 session which does not 08:10.200 --> 08:17.000 feel good to use. After we switch away Lanto you can see we have fully accelerated OpenGL and 08:17.000 --> 08:25.640 Vulcan so you can even enjoy some VK Quake and not just Synodic. So the conclusion of this experiment. 08:25.640 --> 08:35.640 This reason stars are terrible but like I tried another one I tried Fedora which also didn't go well 08:36.520 --> 08:43.720 but that's something for them to look into. If there's like anything missing from our Uboot 08:43.800 --> 08:51.320 staff configs to make like certain EV wars writable or something, please let us know but I don't think 08:51.320 --> 08:58.840 that is the problem at this point. I'm not super sure. But it's important that like distributions 08:58.840 --> 09:05.240 might want to just test a mainland Uboot setup like that just to make sure they're in 09:05.240 --> 09:10.680 solution making it wrong assumptions because at the end of the day if people go out and try 09:10.680 --> 09:16.520 this and then it doesn't work and are going to be like oh support for this SLC is bad even though 09:16.520 --> 09:22.440 it's actually good and it's just installed tripping over itself and that doesn't feel very nice when 09:22.440 --> 09:30.760 you put a lot of work in. But yeah you can move from it just funny just works you don't need anything 09:31.480 --> 09:38.840 fancy device specific images become redundant. So let's go into what kind of change. 09:39.400 --> 09:45.000 During this year and one big topic was molecule abuse and you say well Molly's last 09:45.000 --> 09:53.160 design why we bring it up will be there is it's no coincidence that when we have the 09:54.120 --> 10:01.160 compliance test suit passing it will pass on our pay 3588 because that is the platform used for 10:01.160 --> 10:07.800 developing the new pan store kernel driver for and that's what pan for us to pan became a user space 10:07.880 --> 10:17.880 driver's target and that whole stack gained Vulcan 1.4 conformance this year it officially passes 10:17.880 --> 10:25.000 the test we officially have permission to use the trademark insert it also passes open feel yes 3.1 10:25.000 --> 10:35.560 conformance that is new this year. Then we start the out the year getting an HDMI output on our 10:35.640 --> 10:45.000 K3588 fairly quickly later during the year we also gained video output for our K3576 and it's up to 10:45.000 --> 10:54.200 HDMI 1.4b right now but HDMI 2.0 2.1 is still a working progress part of it has already been 10:54.200 --> 11:01.080 posted to list part of it is still in development and yeah this includes HDMI audio and CEC. 11:01.400 --> 11:07.160 Display port this cool because some of whom rock ship actually wrote a new main land driver 11:07.160 --> 11:11.800 from for this it's not just the dance stream copy base it's been cleaned up it was actually a new 11:12.520 --> 11:20.120 gear and bridge driver for it. It's a merge track name or arcade with 588 I think just today or yesterday 11:20.120 --> 11:28.200 there's a middle day new region for arcade 3576 support so it's working progress on this but 11:28.200 --> 11:35.080 cusbc alt mode has some complications because cusbc is a whole complicated mess and of course 11:35.080 --> 11:40.040 come forget me pdsi that's usually used for like embedded displays and stuff like that 11:40.840 --> 11:49.400 this also supported for both of those ssc's now we didn't put so the non s or k3588 has actually 11:49.400 --> 11:57.720 hmi inputs and we've upstreamed the video for Linux based driver for that it doesn't do audio yet I don't think 11:57.720 --> 12:06.040 and it doesn't do 10 bit yet I don't think but you can totally just like capture hmi on the board 12:06.040 --> 12:11.000 and you can like set your own custom edits and make it pretend to be some monitor or whatever 12:12.600 --> 12:20.120 then we have if you attended the talk I think yesterday we have new me pcs ad drivers for various 12:20.120 --> 12:28.360 structured ports that work was done by Michael Rich it's partially merged work is still ongoing 12:28.360 --> 12:35.800 it's a really complex area of work and if you like heard the thoughts before this also involves 12:35.800 --> 12:42.440 hardware ISPs and we don't have a driver for those yet things are being worked on in that direction 12:42.440 --> 12:48.520 but until then you can use the software ISP to avoid having a green picture 12:50.360 --> 12:55.240 and you was another big edition this year 12:55.240 --> 13:03.880 to me we chose a 10 bit as on his own in his own time he wrote rocket which is a new mainline driver 13:03.880 --> 13:12.440 for the npu funnel in the or k3588 this is an inference and few several generations removed from the open source 13:12.440 --> 13:19.720 and media laid design the user space is implemented as like a teflon delegate in maza so you just 13:19.800 --> 13:26.840 pick up three maza it has that software in it the kernel side is using the new DRM Axel subsystem 13:28.440 --> 13:34.280 the other stuff SLC is released around that time I have very similar and few designs 13:34.280 --> 13:40.520 but they're not precisely the same and I did have some difficulty trying to get it make it work 13:40.520 --> 13:44.840 for our k3586 but at the time I didn't have a register map for the npu there 13:45.800 --> 13:53.640 I assume it won't take as much work as the initial driver to support those but it does still need 13:53.640 --> 13:56.680 work so it's not like a just problem play kind of thing 13:59.320 --> 14:05.000 multimedia is another great area that's a lot of stuff during this year 14:06.360 --> 14:11.080 the hardware we did the coding drivers on this in mainline use the beautiful links to request 14:11.080 --> 14:17.240 stateless decoding interface this is a cross vendor u api this is nothing to do with the 14:17.240 --> 14:22.440 rock chip mpp library so the ffnp licensing drama is not relevant 14:24.120 --> 14:31.720 we started a year out of already having a via one decoding on our k3588 that's a gave you one decoder on 14:31.720 --> 14:37.560 there is a license design so other licensees of that hardware design will also profit from that 14:37.560 --> 14:46.360 if they decide to move up stream then that left us an over route a decoder driver for the 14:46.920 --> 14:56.280 h264 and h265 horror decoders which are in-house rock chip designs this was merged just recently 14:57.080 --> 15:03.320 I think I wrote your first AK decoder driver in mainline I think Qualcomm might have beaten us by one release 15:03.400 --> 15:08.040 but we can still claim we're the first stateless 8k decoder in mainline 15:09.640 --> 15:16.120 and also I saw some people were like playing with using those drivers on our k3588 and 66 15:16.120 --> 15:19.880 so I shouldn't be too much more work to get those working on those two chips 15:21.960 --> 15:27.640 and yeah the ffnpx side of things is handled by Jonas Pibu Kormann who has been 15:27.640 --> 15:34.120 as I mean that for review that has been quite a long story getting the support for that API there 15:35.000 --> 15:37.560 and upstream this stream of food already is supported 15:39.720 --> 15:46.040 then a lot of stuff happened with our k3588 support we basically started the year out not being able to 15:46.040 --> 15:51.960 really boot that and now a lot of stuff works but first like why even bother 15:52.680 --> 15:59.800 yeah it is all the license design but if you look more closely they actually operated a lot of 15:59.800 --> 16:07.720 stuff on the like SSE peripherals side so they have a new audio controller driver they have 16:07.720 --> 16:15.560 UFS now new PWM hardware etc etc they also have a new license interconnect which has DPCA coherence 16:15.560 --> 16:21.640 extensions so you can just slam an AMD GPU in there no kernel patches needed it just works 16:22.280 --> 16:28.920 you only get PCA to one lane but that's actually how I've been doing some AMD GPU work at work 16:28.920 --> 16:34.920 because I already had my ok service and 6 ports set up for net booting a kernel on each boot so 16:34.920 --> 16:44.920 why not use that and yeah it's a good mix of like being kind of closed or k3588 where we can 16:45.000 --> 16:51.880 pick a pack of some work but it has enough new stuff to where we can get prepared for new 16:51.880 --> 16:57.880 or chips and kind of smoothly transition into supporting those as well so basically as I 16:57.880 --> 17:04.520 see support lenses this year which is like clock 3 so it's all that good stuff and a lot of 17:04.520 --> 17:09.320 peripheral drivers needed basically no adjustment because they burn change at all and it's a lot of 17:09.320 --> 17:18.440 license stuff then the UFS driver was actually upstream by rocket themselves which was cool 17:18.440 --> 17:24.520 to see for those model where that's a new flash source standard it's fairly common in phones now 17:24.520 --> 17:32.680 I think it's kind of replaces EMMC and it's basically you get NVME speeds but they very 17:33.160 --> 17:40.840 look kind of small footprint then we have a new fire driver for USB display port because of course 17:40.840 --> 17:49.080 display port now often goes over USB connectors audio driver has been upstream by me I did that 17:50.440 --> 17:57.240 and yeah I needed to blockchain the clock controller a bit for that arm trusted firmware that's good 17:57.240 --> 18:05.240 because with upstreaming of drone trusted firmware code it's the only blob that remains is 18:05.240 --> 18:11.720 the DDRM training blob that runs once a boot and the rest of the stack is fully open 18:13.640 --> 18:18.520 Sebastian Raquel who is also here in the audience today has been going around 18:18.520 --> 18:23.320 squashing all of suspend resume issues I think okay 3586 should now suspend resume fine 18:24.040 --> 18:30.120 okay 3588 I think not yet but I'm also way on reselling him here because whenever there is a 18:30.120 --> 18:35.320 complex issue he's basically the person handles it again you see rock chip actually contributes 18:35.320 --> 18:40.360 they are upstreaming can support which is interesting for cars industrial automation 18:41.720 --> 18:48.200 3D printing now I think and yeah also roads two hardware RNG drivers because they decided to 18:48.200 --> 18:57.560 change up the hardware the newer driver the RK 3576 one doesn't yet make use of all the features 18:57.560 --> 19:03.640 of the hardware but neither does the downstream driver so I don't feel too bad about it it works 19:05.640 --> 19:10.920 and yeah one of the big things still missing is Peter William output and capture I'm still working 19:10.920 --> 19:16.600 this time allows it's posted on the list I need to iterate it on it's some more but yeah it's 19:17.160 --> 19:26.520 a lot of fun and the Peter William work also made me re-factor a lot of rock chip drivers 19:26.520 --> 19:32.840 in the like they had a hybrid update macro that was copy-pasted all across like the Linux 19:32.840 --> 19:38.680 kernel and the unified all of those and accidentally north-sniped Linux Torvals into coming up with 19:38.680 --> 19:45.080 cursed replacements for the Gen Mask macro for like the next two weeks which was extremely good to see 19:47.000 --> 19:53.560 going to give you a few flat out again you're not a squibble carman just doing amazing work 19:53.560 --> 20:00.760 getting your boots to boot on these ports mainline you boot that is even on like this smaller ones 20:00.760 --> 20:05.880 well like the okay revivo 6 which is an interesting as a seat to look out for if you are into 20:05.880 --> 20:13.320 very cheap stuff I got entrepreneur who is the mainline maintainer for rock chips off systems 20:13.960 --> 20:22.920 is just doing amazing work it does a great job at just always being on time I guess 20:23.960 --> 20:30.840 we rarely have problems with having a missed full request and stuff like that then we have a lot of 20:30.840 --> 20:35.480 rock chip employees who contribute now you can see all those names I want to try to pronounce them 20:35.480 --> 20:41.480 all because I will horribly boot for them but I want to give a special shout out to dumb 20:42.120 --> 20:48.760 who actually during like my PWM upstream he replied to me and explained some part of the hardware 20:48.760 --> 20:53.320 to me because he was literally the guy who wrote the downstream driver and I think probably 20:53.320 --> 21:00.200 wasn't both in the hardware design and yeah I also want to give you a shout out to thanks 21:00.200 --> 21:06.760 12 for working on VR putting you boot for okay 3668 because as I mentioned that will make a lot 21:06.760 --> 21:13.960 of the user experience mover when you use generic images looking at the future 21:18.040 --> 21:24.760 yeah I think okay 3586 is going to land this year I think that's going to become bootable it 21:24.760 --> 21:31.640 depends on where they get the matrix IO upstream which is like the new way of having you will be able 21:31.640 --> 21:38.200 to mobs every pin function to every other pin function every other pin and yeah I think you 21:38.200 --> 21:41.640 boot needs to market itself better because a lot of people don't seem to realize you can just use 21:41.640 --> 21:47.640 main land you boot and use it as a platform firmware and boot QEFI images just the old thing 21:47.640 --> 21:53.880 you need to use like the vendors device specific images the video field for Linux requests 21:53.880 --> 22:00.840 say it is important in UAPI is a big on-resolved question but I think there's a movement on that front 22:01.880 --> 22:06.920 we should move up ship to the generic interconnect framework because right now I'm not sure 22:06.920 --> 22:13.320 how well the QOS is handled I vaguely remember it's not handled very well and on for example 22:13.320 --> 22:21.720 okay 3328 which is don't have video output control or QOS so if the memory bandwidth is under strain 22:21.720 --> 22:27.400 you just get video glitches because it doesn't have to bandwidth and yeah I'm hoping you'll see more 22:27.400 --> 22:33.480 camera things in 2021 and there is a lot to do we have a status matrix for the two SSEs we 22:33.480 --> 22:40.280 actively work on there are so more capabilities faster than six drivers that I assume will work 22:40.280 --> 22:47.400 on these other SSEs most of these aren't very interesting SSEs but at least to me but they use the same 22:47.400 --> 22:54.920 kind of order designs and of course please do write device 3s for your device against main 22:55.000 --> 23:01.640 i'm bannings questions yes 23:17.320 --> 23:22.760 about secure boot yes so some of the reverse engineer has a cool reboot works on RK 3588 23:23.720 --> 23:30.200 officially I think rock chip makes this on an NDA for that kind of OTP fuse info 23:30.840 --> 23:37.240 it's one of a few things today do require an NDA for RK 3566 we don't have anything yet 23:37.240 --> 23:42.440 I assume it's not going to meet too much different so no secure boot yet there is an unofficial way 23:42.440 --> 23:47.800 someone has made it work and that's so far yeah well I can say about it 23:48.120 --> 23:52.520 yes 23:52.520 --> 24:00.520 did you try out the UDK 2 firmware for the RK 3588 the EDK 2 TNO Core firmware for that 24:00.520 --> 24:07.320 um I have seen it I don't like it because it's EDK 2 which means it's not a code 24:07.320 --> 24:14.680 base I like very much and it's a fork of EDK 2 which means like all of this work is going to 24:15.640 --> 24:22.280 kind of shamblon in a device specific for that in 10 years I don't know if it's going to be 24:22.840 --> 24:28.120 still maintained the good part about you would is that since it has the same lessons as Linux 24:28.120 --> 24:33.720 we can borrow a lot of driver codes from Linux itself and because it's all all device support 24:33.720 --> 24:42.920 is in the same repository and everyone licenses the same to you like USB controller hardware and 24:43.000 --> 24:46.680 stuff like that you have a lot of like codes that doesn't need to be double-cated 24:56.280 --> 24:59.080 we're good 24:59.400 --> 25:05.160 is the 3.06 which is the option that said there they have it may not be the same yet so 25:07.160 --> 25:15.480 yeah so 3.506 that hasn't been fully mainland yet there is rock ship is working on it 25:16.360 --> 25:19.880 some other from other people from the community are working on it as well 25:21.480 --> 25:27.800 I think you can already do mainland UBoot with a onlist patch set by Kribu 25:28.680 --> 25:33.880 however kernel will take a bit more there's currently some difficulty figuring out how 25:33.880 --> 25:40.600 we do the deep findings for the new rock ship matrix I.o but definitely expected to happen this year 25:41.320 --> 25:45.000 and I would say they'll look out for that 25:47.800 --> 25:51.240 okay thank you 25:57.800 --> 26:00.120 you