WEBVTT 00:00.000 --> 00:10.000 I think you just need to prep the power and say a hum or something. 00:10.000 --> 00:12.000 Just get started, people will quite soon. 00:12.000 --> 00:14.000 Okay, 13 hundred layers. 00:14.000 --> 00:18.000 Top begins. 00:18.000 --> 00:21.000 Greetings, salutations. 00:21.000 --> 00:25.000 My self and hands will be giving you a talk on a live camera, 00:25.000 --> 00:29.000 and specifically a software ISP in the camera. 00:29.000 --> 00:31.000 This is Hans. 00:31.000 --> 00:33.000 I'm Brian. 00:33.000 --> 00:35.000 As you may know. 00:35.000 --> 00:37.000 So, headways say your name exactly. 00:37.000 --> 00:38.000 Hans the corner. 00:38.000 --> 00:39.000 Hans the corner. 00:39.000 --> 00:40.000 The corner. 00:40.000 --> 00:42.000 But it's just Hans the good also worked. 00:42.000 --> 00:44.000 I think the good the better the ugly. 00:44.000 --> 00:46.000 Actually I saw your full name. 00:46.000 --> 00:48.000 It's much bigger than that, actually. 00:48.000 --> 00:49.000 Very loud. 00:49.000 --> 00:52.000 You say my name just like Brian. 00:53.000 --> 00:56.000 We both worked in the camera domain. 00:56.000 --> 01:01.000 I'm doing some stuff with the welcome camera at Iris the Venus. 01:01.000 --> 01:06.000 I do pull a request now for various people. 01:06.000 --> 01:11.000 And I've had that experience for the last few kernels, 01:11.000 --> 01:17.000 where I've done side off by a hundred and fifty patches on which are mine. 01:17.000 --> 01:21.000 So, be careful what you wish for when you wish for it to be a kernel container. 01:21.000 --> 01:24.000 This is what the job really is. 01:24.000 --> 01:25.000 You know that. 01:25.000 --> 01:27.000 That feeling as well. 01:27.000 --> 01:32.000 So, our talk today will be basically on software ISP. 01:32.000 --> 01:34.000 What an ISP is. 01:34.000 --> 01:36.000 Why you need an ISP. 01:36.000 --> 01:39.000 And what you do when you don't have an ISP. 01:39.000 --> 01:41.000 Up to the mind says. 01:41.000 --> 01:48.000 So, first of all, I go to assume that you're all very intelligent people who may know absolutely 01:48.000 --> 01:54.000 nothing about cameras or images or image processing or anything like that. 01:54.000 --> 02:00.000 So, you're going to learn very quickly because I was trying to estimate the average IQ in this room. 02:00.000 --> 02:03.000 And I would say it's probably not less than one-third. 02:03.000 --> 02:07.000 So, we'll all be very quick in the uptake here. 02:07.000 --> 02:09.000 Bear encoding. 02:09.000 --> 02:10.000 You may have heard of it. 02:10.000 --> 02:16.000 It was created by mine called Bruce Bear. 02:16.000 --> 02:26.000 If I remember his name correctly, in 1974, according to this document right here. 02:26.000 --> 02:32.000 And the theory of operation here is you may notice this mosaic pattern works. 02:32.000 --> 02:37.000 Green blue, green blue, green blue, red green, red green, red green, red green, red green, red green. 02:37.000 --> 02:42.000 You might say to yourself, why does green appear so many times? 02:42.000 --> 02:50.000 And the answer is actually because if you look at this graphic here, your AI is most sensitive to the 02:50.000 --> 02:54.000 colors spectrum and green whether or not you know that. 02:54.000 --> 03:01.000 So, if you think about how a light sensor actually works, it's just an ADC, right? 03:01.000 --> 03:03.000 It goes from zero to some value. 03:03.000 --> 03:06.000 So, it's just a number. 03:06.000 --> 03:12.000 What you have to do is you have to make that sensor sensitive to one particular color. 03:12.000 --> 03:14.000 So, you put a filter on it. 03:14.000 --> 03:17.000 It's basically a black and white sensor. 03:17.000 --> 03:21.000 You put a filter on it for green or red or green or blue. 03:21.000 --> 03:26.000 And you're recovering the blue part of the spectrum at the blue sensor. 03:26.000 --> 03:31.000 But immediately you already have a problem because, okay, I've captured green right there. 03:31.000 --> 03:33.000 But it didn't really capture blue. 03:33.000 --> 03:39.000 Yeah, actually when you capture green, you still capture elements of blue and you capture elements of red. 03:39.000 --> 03:48.000 What you have to do is you have to infer the value of the red at this green based on the adjacent red. 03:48.000 --> 03:53.000 And infer the value of the blue at this green for the adjacent blue. 03:53.000 --> 03:58.000 That's just a simple process of, let's call, linear interpolation. 03:58.000 --> 04:07.000 Just looking at the adjacent pixels and making it a fairly good educated gas based on the value that you sampled at the pixel of your ad. 04:07.000 --> 04:13.000 And the adjacent values, what the RGB for each one of those pixels was. 04:13.000 --> 04:16.000 And of course, again, because you're always more sensitive to green. 04:16.000 --> 04:21.000 You want to capture as much information in green as you possibly can. 04:21.000 --> 04:26.000 And so, basically what comes out of image sensors isn't an RGB picture. 04:26.000 --> 04:29.000 It's, it's usually this bayering called a picture. 04:29.000 --> 04:34.000 For exactly that reason to capture as much information as possible, 04:34.000 --> 04:36.000 but you can't capture all information in green. 04:36.000 --> 04:39.000 Because obviously then, the image will be green. 04:39.000 --> 04:47.000 Is this bit you, see you, see you, see you, see you, see you, see you, see you, see you, see you, see you, see you, see you, see you, see you, see you. 04:47.000 --> 04:51.000 So, one of the reasons why I got involved in cameras or re-envolved. 04:51.000 --> 04:54.000 I also bought involved in cameras in 2008-ish. 04:54.000 --> 04:59.160 It's because X86 laptops, I work for Qualcomm now, 04:59.160 --> 05:02.040 but it's originally a work for Red Hat and a work for X86 laptops. 05:02.040 --> 05:05.800 X86 laptops used to have USB camera modules. 05:05.800 --> 05:08.920 And USB camera modules use a standard protocol called UVC. 05:08.920 --> 05:11.280 And there are things like V-Mose and E-King, 05:11.280 --> 05:13.920 but also picking the right gain and exposure value 05:13.920 --> 05:15.240 adjusting the white balance. 05:15.240 --> 05:17.440 We're all done by a little micro controller 05:17.440 --> 05:23.840 inside the camera module. 05:23.840 --> 05:25.440 So it pretty much just worked. 05:25.440 --> 05:28.320 You could just ask the USB camera module 05:28.320 --> 05:29.720 give me a picture of this resolution 05:29.720 --> 05:32.480 and it would give you ready to do pictures. 05:32.480 --> 05:34.800 But recently, also X86 laptops 05:34.800 --> 05:40.000 and this Qualcomm laptop here also have moved to the smartphone 05:40.000 --> 05:42.240 camera model, which is you just have a roll sensor. 05:42.240 --> 05:44.240 And you actually get a roll sensor or images. 05:44.240 --> 05:46.440 So roll buyer encoded images. 05:46.440 --> 05:50.000 Just like the roll images, if you have a professional camera 05:50.000 --> 05:53.800 and you decide to pick a roll images in Linux, 05:53.800 --> 05:56.240 and then we need to do something with those roll images 05:56.240 --> 05:59.760 and make them usable for a video conferencing or whatever. 05:59.760 --> 06:01.920 This is where our actually lip camera comes in, 06:01.920 --> 06:03.960 because this is a complex problem. 06:03.960 --> 06:06.200 There are a bunch of building blocks in hardware, 06:06.200 --> 06:09.480 which you can help to translate a roll image into a proper image, 06:09.480 --> 06:11.800 which together we usually call an ISP. 06:11.800 --> 06:14.560 A hardware image signal processor. 06:14.560 --> 06:16.920 ISP is tend to be shrouded in sequencing, 06:16.920 --> 06:20.080 that it's secrecy, that's slowly changing. 06:20.080 --> 06:22.920 But the reason why the soft ISP was created 06:22.920 --> 06:25.560 is to just have a generic software implementation, 06:25.560 --> 06:27.160 which would work everywhere, 06:27.160 --> 06:29.200 why we were working on figuring out 06:29.200 --> 06:31.000 how to actually use hardware ISPs. 06:31.000 --> 06:34.560 So that is where the software ISP comes in. 06:34.560 --> 06:38.160 So first, what does the hardware ISP do? 06:38.160 --> 06:41.760 It's basically dedicated ASIC blocks, 06:41.760 --> 06:44.920 which are specialized in doing the necessary transformations 06:44.920 --> 06:46.080 really efficiently. 06:46.080 --> 06:48.960 So really fast, but also really power efficient. 06:48.960 --> 06:51.240 It does the demo's E-king. 06:51.240 --> 06:53.640 The slide says it's implements the three ASIC. 06:53.640 --> 06:56.400 So in camera, as we often talk about the three ASIC, 06:56.400 --> 06:58.760 which is auto gains like exposure. 06:58.760 --> 07:01.440 Exposure time is very important to get a good picture, 07:01.440 --> 07:04.080 and then sometimes you also need to boost the gain a bit 07:04.080 --> 07:05.320 if it's dark. 07:05.320 --> 07:07.440 Even after the exposure time, auto white balance 07:07.440 --> 07:10.880 and auto focus if you have a non-fixed focus lens. 07:10.880 --> 07:13.680 It says here, it implements the three ASIC, 07:14.680 --> 07:16.760 it's nice for on the slide, it's not correct. 07:16.760 --> 07:18.040 It implements statistics. 07:18.040 --> 07:20.640 It does it gives you a bunch of histograms over the role bear, 07:20.640 --> 07:23.120 and those help can help you to figure out 07:23.120 --> 07:25.760 what's the right exposure value, et cetera, is. 07:25.760 --> 07:29.160 And then there is more secret in there. 07:29.160 --> 07:34.120 There is a more simple ISP example diagram at the top. 07:34.120 --> 07:36.760 You see the role input image that is something 07:36.760 --> 07:39.080 called black level correction, because if you actually 07:39.080 --> 07:42.880 fully cover a sensor, so it doesn't get any light on it at all, 07:42.880 --> 07:44.840 even if you could like a really thick cover, 07:44.840 --> 07:47.000 it will not give you zero pixel values. 07:47.000 --> 07:49.440 It will usually give you something about a value 07:49.440 --> 07:54.120 of around 16 for if the values go up to 25 or 64, 07:54.120 --> 07:55.920 if they go up to 1,000, 24 or whatever. 07:55.920 --> 07:59.160 So there is like this black level, which also has to do 07:59.160 --> 08:02.080 with how the analog electronics works and bias currents, 08:02.080 --> 08:04.520 which you need to basically subtract from all the pixel values 08:04.520 --> 08:07.280 to make black actually be black. 08:07.280 --> 08:09.560 Then that pixel correction, there might be some defective pixels. 08:09.560 --> 08:13.160 You want to sort of interpolate around those, 08:13.160 --> 08:16.400 and ignore them, and then we get the demos 08:16.400 --> 08:18.800 in eking and in white balancing, and some other stuff, 08:18.800 --> 08:19.720 like denoising. 08:19.720 --> 08:21.320 This is from the work chip ISP. 08:21.320 --> 08:25.280 You see that the actual physical hardware ISP is 08:25.280 --> 08:27.800 a lot more complicated, so there are a bunch of other things 08:27.800 --> 08:31.240 which hardware ISP does do. 08:31.240 --> 08:34.000 So we have the first software ISP, which we 08:34.000 --> 08:37.760 presented here two years ago, with this exact same room. 08:37.760 --> 08:39.240 Back then, we just had a demo. 08:39.240 --> 08:41.520 But it was merged a couple of months later, 08:41.520 --> 08:43.560 is the CPU ISP. 08:43.560 --> 08:46.440 I'd use this a little camera, because the camera is quickly 08:46.440 --> 08:48.240 becoming or actually is pretty much an hour 08:48.240 --> 08:51.400 day to de-standard for dealing with cameras in Linux. 08:51.400 --> 08:53.800 We no longer want applications to directly access 08:53.800 --> 08:57.280 the video hardware notes directly, because that works fine 08:57.280 --> 08:59.320 with the UVC camera, where an application could use, 08:59.320 --> 09:02.640 say, give me a picture with a modern complex pipeline. 09:02.640 --> 09:05.160 You need to set up and configure the whole pipeline. 09:05.160 --> 09:09.000 So we don't want each application to have to do that. 09:09.000 --> 09:12.040 So the software ISP, which has been implemented 09:12.040 --> 09:16.680 and that streams does some of the statistics stuff. 09:16.680 --> 09:18.560 And then it does outdoor white balance and outdoor game 09:18.560 --> 09:19.880 in camera correction. 09:19.880 --> 09:21.680 It doesn't support outdoor focus yet. 09:21.680 --> 09:23.520 There are some patches out of three for that. 09:23.520 --> 09:25.240 Let's not dare yet. 09:25.240 --> 09:29.680 So the software ISP works pretty well on powerful hardware. 09:29.680 --> 09:32.400 And it works pretty poorly on, on, say, 09:32.400 --> 09:35.720 a Raspberry Pi 3 level device, like the Arduino Q, 09:35.720 --> 09:37.440 which was recently released. 09:37.440 --> 09:41.200 And it does six FPS there with an IMX-2-18 sensor, 09:41.200 --> 09:43.400 and four resolution, which is not what we want, 09:43.400 --> 09:45.600 because the sensor can do 30 FPS. 09:45.600 --> 09:46.600 So yeah. 09:46.600 --> 09:49.800 So this is where Brian's work from the past one 09:49.800 --> 09:51.120 and a half year of something comes in, 09:51.120 --> 09:53.400 who has been working on GPU acceleration 09:53.400 --> 09:55.360 for the software ISP. 09:55.360 --> 10:02.720 So like Hans mentioned there, on the class of device, 10:02.960 --> 10:08.080 this laptop here in the T14S, running the home OS system, 10:08.080 --> 10:11.480 the X-180, 100, you're more than enough processing 10:11.480 --> 10:17.760 power to do all of the 3A's in the CPU. 10:17.760 --> 10:21.520 However, when you start going down to just slightly lower 10:21.520 --> 10:26.280 down, so there's a type of processor from Qualcomm called Kona, 10:26.280 --> 10:29.480 which is released in 2019, the SM8250. 10:29.480 --> 10:32.000 I have that development board in my desk. 10:32.000 --> 10:35.120 And I cannot get 30 frames per second 10:35.120 --> 10:37.160 with the RGB sensor used in CPU ISP. 10:37.160 --> 10:39.400 Now this is a powerful processor. 10:39.400 --> 10:42.200 So as we go further down into the, 10:42.200 --> 10:46.880 embedded domain there, you simply just can't do, 10:46.880 --> 10:52.480 even the basics, the 3A's in CPU. 10:52.480 --> 10:55.760 However, there's this magic device called the GPU, 10:55.760 --> 11:00.760 and the GPU is actually a massively parallel pixel processing engine. 11:00.760 --> 11:07.360 It's all purpose in life as to process pixels in parallel. 11:07.360 --> 11:10.640 And this has been useful in LLMs, and so on. 11:10.640 --> 11:12.040 Actually, I'm not going to talk about that. 11:12.040 --> 11:15.120 I don't really know how the massively parallel bit of it works, 11:15.120 --> 11:18.760 but definitely the pixel parallel bit of it works really well. 11:18.760 --> 11:23.320 So for our use case, that turns out to be quite interesting 11:23.320 --> 11:26.240 because as I was telling you before, 11:26.240 --> 11:32.960 the first thing we need to do is interpolation of adjacent pixels. 11:32.960 --> 11:36.000 So if you think about it, your fragment shader 11:36.000 --> 11:39.640 is basically a program that gets run for one pixel, 11:39.640 --> 11:43.160 and you can decide stuff to do at that pixel. 11:43.160 --> 11:47.520 Well, if you have to linearly interpolate data, 11:47.520 --> 11:49.480 that's great because I'm at one pixel. 11:49.480 --> 11:50.880 I can look at the next pixel. 11:50.880 --> 11:53.000 I can look at the pixel to the other side, 11:53.000 --> 11:57.120 and then I could decide to perform logic based on that. 11:57.120 --> 11:58.360 And I'm doing it in parallel. 11:58.360 --> 12:02.240 So I'm like a scalar CPU, even the superscaler CPU, 12:02.240 --> 12:05.800 which has loops, the whole purpose of the GPU 12:05.800 --> 12:08.920 is to kind of execute a whole bunch of tasks, 12:08.920 --> 12:11.880 a whole bunch of tile tasks in parallel. 12:11.880 --> 12:16.320 So that's really how GPU ASP comes into its own there. 12:16.320 --> 12:20.560 And that the camera already had some of the upstream 12:20.560 --> 12:24.400 and some of you in the narrow actually, where I have to work. 12:24.400 --> 12:29.840 Obstringed support for vertex fragment shader 12:29.840 --> 12:32.880 to do a simple debaring. 12:32.880 --> 12:37.040 However, it didn't have any of the necessary additional steps, 12:37.040 --> 12:41.800 so black level or contrast or color correction matrices, 12:41.800 --> 12:44.280 or any of that others solve. 12:44.280 --> 12:48.160 So what they spoke to Kirin and Laurent 12:48.160 --> 12:57.560 that over a number of months was let's reuse the existing pixel 12:57.560 --> 12:59.400 and fragment shaders. 12:59.400 --> 13:01.720 Take it out of the QT application where it lives 13:01.720 --> 13:06.800 and brings it into lib camera, proper, headless, 13:06.800 --> 13:09.680 which is what we did. 13:09.680 --> 13:13.880 And you might ask, well, what you use fragment and pixel shader. 13:13.880 --> 13:16.480 And the first size is that is because we already have it. 13:16.480 --> 13:17.920 And we want to just reuse software. 13:17.920 --> 13:20.600 And there's no use throwing out software that works. 13:20.600 --> 13:26.920 It represents development effort that yielded results. 13:26.920 --> 13:30.000 And the second answer is actually, once you get through implementing it, 13:30.000 --> 13:33.760 you find that for most of the stuff that you actually 13:33.760 --> 13:37.320 for one specific thing you want to do, specifically 13:37.320 --> 13:40.840 the stuff Robert Mader from Calabra, worked on Robert's right 13:40.840 --> 13:43.720 here, given you shared it, because Robert did some great work 13:43.720 --> 13:47.480 with this, but to do scaling, actually, 13:47.480 --> 13:49.840 you can do that in Mesa quite well. 13:49.840 --> 13:51.560 You can do it quite handily. 13:51.560 --> 13:54.640 And at this stage, actually, you don't want to compute shader. 13:54.640 --> 13:59.080 You really do want a proper Mesa program for that. 13:59.080 --> 14:07.800 So we're just going to talk a little bit about lens shading later on. 14:07.800 --> 14:10.000 Dead pixel correction, denoising. 14:10.000 --> 14:12.240 I mean, these are the next kind of steps. 14:12.280 --> 14:13.600 So the image is noisy. 14:13.600 --> 14:14.440 You'll see it later on. 14:14.440 --> 14:16.160 It's noisy. 14:16.160 --> 14:19.200 And we're saturating in places. 14:19.200 --> 14:22.080 But anyway, it runs really, really fast in comparison 14:22.080 --> 14:25.840 to CPU ASP for every, so let's actually move on to next slide. 14:25.840 --> 14:29.120 So this is the implementation we have right now. 14:29.120 --> 14:32.760 It was committed to not 0, 6. 14:32.760 --> 14:36.960 So yeah, well, this is upstream and in-depth camera now. 14:36.960 --> 14:41.200 We managed to merge this just before Christmas. 14:41.200 --> 14:43.040 So this is a very simple. 14:43.040 --> 14:47.520 So before we had the rupture, yeah, this really complex 14:47.520 --> 14:50.000 and kind of cool ASP. 14:50.000 --> 14:57.400 So in homage to that very nice graphic we have, this one here. 14:57.400 --> 14:59.480 But this is basically what we're doing right now. 14:59.480 --> 15:05.000 We take the CSI2, Mippy Bayer data in, D-Mosec, 15:05.000 --> 15:09.200 black level correct, anybody who knows anything about this, 15:09.200 --> 15:11.000 which would basically be the lip camera guys 15:11.000 --> 15:12.840 will tell me, this is already wrong. 15:12.840 --> 15:17.240 You should be black level correcting app before you 15:17.240 --> 15:21.880 D-Mosec, but we do it straight after that will be fixed. 15:21.880 --> 15:25.000 We apply a color correction matrix. 15:25.000 --> 15:27.120 We tap the contrast and then we apply the camera. 15:27.120 --> 15:30.080 And what we get is, of course, RGBA, H-A, H-A, H-A, 15:30.080 --> 15:37.240 H-A, because the GPU view of the world, 15:37.240 --> 15:40.080 the end of its pipeline is always to write a D-word. 15:40.080 --> 15:43.720 It always wants to write 32 bits in a fragment shader. 15:43.720 --> 15:46.000 So with this model right here, 15:46.000 --> 15:51.560 it will not be possible to easy color space conversion. 15:51.560 --> 15:53.880 However, multi-pass. 16:00.520 --> 16:04.960 There it is, it's good and she's awesome. 16:04.960 --> 16:10.200 And obviously look best on, is the best French director ever. 16:10.200 --> 16:12.800 So then the next steps from G-P-U-I-S-P will be this. 16:12.800 --> 16:16.640 We will have a multi-passed through. 16:16.640 --> 16:19.840 So we'll go from the Bayer domain to the RGB domain. 16:19.840 --> 16:22.960 This is the second phase here. 16:22.960 --> 16:24.760 We will scale in Sharpen. 16:24.760 --> 16:30.760 At this stage here, you can see we'll go from a not vertex shader 16:30.760 --> 16:33.880 to a non-not vertex shader, and this is where 16:33.920 --> 16:35.560 we're going to scale. 16:35.560 --> 16:37.360 We use Robert's code here. 16:37.360 --> 16:38.920 Oh, sorry, I might have framed. 16:38.920 --> 16:40.320 Oh, it shouldn't go past the black line. 16:40.320 --> 16:42.440 They can't see you past the black line. 16:42.440 --> 16:49.480 So this projection matrix is where you will scale. 16:49.480 --> 16:50.680 And we'll do some sharpening. 16:50.680 --> 16:51.880 And then we will play Gamma. 16:51.880 --> 16:54.880 So if you don't just hear in the previous slide, 16:54.880 --> 16:57.040 we will play Gamma at the end. 16:57.040 --> 17:02.080 Here we will play Gamma in the last vertex fragment shader. 17:02.080 --> 17:05.160 Why does Gemini tell me that's where it should be applied? 17:05.160 --> 17:07.440 And it knows more than me. 17:07.440 --> 17:10.120 And then finally, we'll move on to a compute shader 17:10.120 --> 17:13.360 to do the pixel format conversion. 17:13.360 --> 17:14.640 And you last play. 17:14.640 --> 17:18.600 And the answer is, because here, the vertex fragment shader 17:18.600 --> 17:22.080 view of the world is to write 32 bits at the end of D 17:22.080 --> 17:25.080 words, both a compute shader, and also 17:25.080 --> 17:27.320 to do non-D word. 17:27.320 --> 17:29.320 So the compute shader is the place that we have to do 17:30.040 --> 17:32.800 pixel format conversion. 17:32.800 --> 17:33.960 And watch the fifth element. 17:33.960 --> 17:36.440 It's an awesome movie. 17:36.440 --> 17:38.880 Next steps are going through this really quickly. 17:38.880 --> 17:42.600 So I just call it basically, what the next steps for GPU 17:42.600 --> 17:46.040 is multi-pass. 17:46.040 --> 17:49.840 Probably is few additional steps as possible. 17:49.840 --> 17:53.160 But you still have to have an RGB behavior domain, 17:53.160 --> 17:56.480 a Bayer domain, and RGB domain. 17:56.480 --> 18:03.880 And most likely, a pixel format conversion phase. 18:03.880 --> 18:06.640 That pixel corrections is probably easy to do. 18:06.640 --> 18:10.240 Other focus, I have no idea, either easier 18:10.240 --> 18:12.640 hard, lens-player correction. 18:12.640 --> 18:17.880 I have no idea how difficult or non-difficult that will be. 18:17.880 --> 18:20.320 And we have a demo. 18:20.320 --> 18:20.840 Yeah. 18:20.840 --> 18:25.560 So last six months, I had a student group 18:25.560 --> 18:30.680 from the Fahalchula in Delft, the Hasselg Scho, 18:30.680 --> 18:33.320 a computer science department working on Lenshade 18:33.320 --> 18:34.200 correction. 18:34.200 --> 18:38.120 And these are two example images of what they have written, 18:38.120 --> 18:40.160 and which is available in the mailing list now. 18:40.160 --> 18:41.440 So if you look at this image, you 18:41.440 --> 18:43.080 can look at the top left and top right corner. 18:43.080 --> 18:44.960 You see it's a bit gray there, while the ceiling 18:44.960 --> 18:46.600 is pretty much white. 18:46.600 --> 18:48.240 And if we now switch to the next one, which 18:48.240 --> 18:50.520 has the LSC applied, the corners 18:50.520 --> 18:52.680 are brightened up quite nicely. 18:52.680 --> 18:55.400 So this is actually quite nice to see that it's doing what 18:55.400 --> 18:58.560 it's supposed to do, because I mean, Lenshade's round. 18:58.560 --> 19:01.880 So it captures not a whole lot of light in the corners, 19:01.880 --> 19:03.680 because well, it doesn't have corners. 19:03.680 --> 19:09.480 So we need to apply a boost to the light level in the corners. 19:09.480 --> 19:12.200 So this is what this does. 19:12.200 --> 19:17.720 So CCMs, as Brian mentioned, and as we'll go in much more 19:17.720 --> 19:21.320 depth in the next talk, the GPU ISP now supports 19:21.320 --> 19:24.080 a color correction matrix. 19:24.080 --> 19:27.160 This allows us to do things like, since the color filter 19:27.160 --> 19:31.320 is leaky, so the blue field, the green cells 19:31.320 --> 19:33.160 get a bit of red and blue light leaked in. 19:33.160 --> 19:38.120 You can subtract red and green from the neighboring, 19:38.120 --> 19:40.160 from it to get the pure green value and things like that. 19:40.160 --> 19:42.560 That is what it's just at 3 by 3 matrix multiplication, 19:42.560 --> 19:44.160 really. 19:44.160 --> 19:45.800 Another thing which it gives us, which is actually 19:45.800 --> 19:47.360 already implemented, but I don't 19:47.360 --> 19:48.600 live in the demo, sorry. 19:48.600 --> 19:50.840 Maybe you can show your, I don't have the saturation 19:50.840 --> 19:51.840 thingy. 19:51.840 --> 19:53.280 I think Karen shows us the next. 19:53.280 --> 19:54.080 OK, yeah. 19:54.080 --> 19:59.160 So if you look at the camera pictures versus, like, 19:59.160 --> 20:01.480 commercial phone application, then the color 20:01.480 --> 20:05.880 pop a lot more in the phone camera apps. 20:05.880 --> 20:08.360 That's also just because they're not really a photo-reelistic 20:08.360 --> 20:09.160 representation. 20:09.160 --> 20:11.440 They boost up what's called saturation, so the color 20:11.440 --> 20:13.120 is look more rich. 20:13.120 --> 20:15.760 And I think we probably need to just, since everyone 20:15.760 --> 20:19.240 has used to that, just put the saturation control default 20:19.240 --> 20:23.240 at, say, 1.2 on the scale of 0 to 2.0 with 1.0 being 20:23.240 --> 20:27.040 neutral, so we'll just actually, the default lip camera output 20:27.040 --> 20:30.360 will also have a bit more rich colors. 20:30.360 --> 20:34.360 A lot more about this in the next thingy, which 20:34.360 --> 20:36.960 is interesting, especially, also, again, we're 20:36.960 --> 20:40.240 by Robert Mader, so for Dora has been shipping the soft ISP 20:40.240 --> 20:44.560 for a while now to support Intel-based NIP cameras, 20:44.560 --> 20:48.160 so the non-liffici cameras on various laptops. 20:48.160 --> 20:50.360 Robert has been working on post-markato as in all the mobile 20:50.360 --> 20:52.680 distros, and this is a small list of phones 20:52.680 --> 20:55.640 where they now have the cameras working using the soft ISP, 20:55.640 --> 20:58.360 especially also with the GPU ISP, because that helps 20:58.360 --> 21:00.880 the lot also on the mobile phones. 21:00.880 --> 21:04.880 And then a demo. 21:04.880 --> 21:10.400 So this is actually the GPU ISP with lens fading correction. 21:14.560 --> 21:21.040 So yeah, it works. 21:21.040 --> 21:33.720 And the color cell aren't actually too bad here in this 21:33.720 --> 21:34.360 light. 21:34.360 --> 21:39.520 So yeah, we have a couple of minutes left for questions, 21:39.520 --> 21:42.640 I hope, it helps if I switch to the, yeah. 21:44.560 --> 21:47.000 Ah, yeah, go ahead. 21:47.000 --> 21:58.080 So you're working in the graphics pipeline, 21:58.080 --> 21:59.440 which is why you're the multi-pass. 21:59.440 --> 22:02.360 Why don't you do everything in a single compute shader? 22:06.360 --> 22:08.360 The question I get all the time, and the answer 22:08.360 --> 22:12.040 is because somebody else already did the really hard work 22:12.040 --> 22:16.840 of decoding, because we don't get a simple blank stream in. 22:16.840 --> 22:21.000 We get a CSI-2, maybe encoded stream in, 22:21.000 --> 22:23.120 which is 10 bits. 22:23.120 --> 22:25.120 So you have to track inside of the 10 bits, 22:25.120 --> 22:28.520 and you're not sampling other specific place. 22:28.520 --> 22:31.360 You're sampling in a thing that slides. 22:31.360 --> 22:34.280 And I don't want to have to understand 22:34.280 --> 22:36.600 that and reimplemented the computer. 22:36.600 --> 22:38.960 And the other answer is, when we started doing it, 22:38.960 --> 22:43.040 we didn't know what generation of GPU 22:43.040 --> 22:45.040 would necessarily try to support. 22:45.040 --> 22:48.720 So we tried to go for the oldest one possible. 22:48.720 --> 22:52.560 And computers came in in the bet 2009, I think. 22:52.560 --> 22:55.200 So I mean, we're casting it that fairly wide, 22:55.200 --> 22:57.800 but the thought when we started doing it was, 22:57.800 --> 23:01.600 we weren't exactly sure how far back in time we could go. 23:01.600 --> 23:04.600 And we want to be able to go as far back in time as possible. 23:04.600 --> 23:07.400 And if you are using a class of hardware that doesn't 23:07.400 --> 23:09.800 support computer shaders, I'm happy to say, 23:09.800 --> 23:13.920 we support you, but maybe get something a bit newer. 23:13.920 --> 23:16.040 Yeah, so basically, we wanted to have like, 23:16.040 --> 23:18.280 open class 2.0 as bottom level, 23:18.280 --> 23:20.640 like anything which opened class 2.0, 23:20.640 --> 23:25.440 to support at least a current single pass, GPU ISP. 23:25.440 --> 23:29.040 Take a little time for one minute and 42 seconds, 23:29.040 --> 23:31.240 more questions. 23:31.240 --> 23:32.400 First up, best rest. 23:32.800 --> 23:37.280 This is really quick. 23:37.280 --> 23:40.040 So it should be less than 30 seconds. 23:40.040 --> 23:42.960 Great to see CSI2 been integrated with the lip camera. 23:42.960 --> 23:47.000 Is there a plan for CSI2 as well going forward? 23:47.000 --> 23:48.880 Security extensions. 23:48.880 --> 23:51.600 That might be a better question for the next talk 23:51.600 --> 23:52.560 because I have no clue. 23:52.560 --> 24:08.680 So when can we expect GPU ISP shipping and post-market OS? 24:08.680 --> 24:13.120 Oh, it's in 0.70, just got tax and release. 24:13.120 --> 24:14.600 So we'd see it up to you. 24:14.600 --> 24:17.720 Aren't you a post-market committer? 24:17.720 --> 24:20.120 It's a question for Robert. 24:20.120 --> 24:22.000 Is shipping? 24:23.000 --> 24:23.680 Today. 24:30.320 --> 24:33.080 So first, thank you very much for your work. 24:33.080 --> 24:37.960 You're like the delighted end of the tunnel for my new X1. 24:37.960 --> 24:42.600 So Fedora and I have a working open source stack 24:42.600 --> 24:46.320 for work with what are Intel webcams. 24:46.320 --> 24:51.200 But do you know, is there any work going on for Debian, 24:51.200 --> 24:56.640 a channel or arch-based history as well? 24:56.640 --> 24:59.640 I guess some people are looking into this. 24:59.640 --> 25:02.560 Sometimes I do get questions because I did the Fedora work 25:02.560 --> 25:06.480 from people for say, Nick, so as that they like, 25:06.480 --> 25:08.360 like, how did you solve this or this in Fedora? 25:08.360 --> 25:11.600 So do you want to make a remark, or? 25:11.600 --> 25:12.600 OK, thanks. 25:12.600 --> 25:18.040 I mean, they come and I think that you don't have a mobile. 25:18.040 --> 25:20.880 OK, so mobile and already has the GPU ISP 25:20.880 --> 25:29.440 even, so I mean, I'm welcome to help any destroyer. 25:29.440 --> 25:32.120 If they have questions, they hit a roadblocks, 25:32.120 --> 25:33.640 but I cannot do all the disrupts. 25:33.640 --> 25:36.040 And my time is up, so thank you all for your time.