WEBVTT 00:00.000 --> 00:23.440 All right, we're going to listen to tracking energy and emissions of user jails by Mahendra. 00:23.440 --> 00:24.440 Thank you. 00:24.440 --> 00:30.920 Good morning, everyone. As my name is Mahendra Pai Puri. Currently, as my day job, I'm working 00:30.920 --> 00:36.880 like a XC and Spoparis. So, yeah, this focus not everything about a political sciences, 00:36.880 --> 00:41.080 like I used to work like one of the national HPC's interests of France, like before this 00:41.080 --> 00:49.080 my current job, and most of this work I've done, like there. So, just to go into the details, 00:49.080 --> 00:54.200 yeah, I don't have to explain this context anymore. The AI servers are like, 00:54.280 --> 01:00.680 consuming the world. Well, actually, this plot, I have to update this, like, because OECD has 01:00.680 --> 01:07.000 updated the numbers and they look even back now. So, yeah, this is one of the, one of the hot problems 01:07.000 --> 01:13.800 that we have right now, and this is something that we need to solve. Well, I mean, within the HPC 01:13.800 --> 01:19.800 platforms, like, yeah, I mean, there's a lot of, there's a lot of solutions that come into to solve 01:19.800 --> 01:23.800 these problems on the system level, like, like optimizing your energy usage of the anti-HPC 01:24.760 --> 01:29.880 but of course, end users are still running like crappy course on the on the HPC platforms, 01:29.880 --> 01:33.720 like, you know, they're just like burning electricity for no reason, right? I mean, even your 01:33.720 --> 01:40.120 system level optimization software can do a lot of things to avoid that, like, the might take on 01:40.120 --> 01:44.200 this problem is like the change as to always come from the people from the end users and, like, 01:44.200 --> 01:49.400 you know, you got to chip the blame on them, like, you know, they had to make them feel guilty, 01:49.480 --> 01:55.480 you know, for, for burning electricity. And that is the only way to, to the practical solutions 01:55.480 --> 02:01.960 to engage them, to optimize their course and better use the, use the HPC resource and, like, 02:01.960 --> 02:07.880 better use energy at the end of the day. And this can be only possible, but, like, if we 02:08.680 --> 02:14.200 actually provide them with the, with the measures and the tools to, to encourage the optimization, 02:14.200 --> 02:18.680 right? I mean, and, like, of course, like, small, so the HPC platforms, like, 02:18.680 --> 02:24.440 are they, they provide very, very, very, very high level metrics, like, not really useful sometimes, 02:24.440 --> 02:28.040 like, they sometimes give some useful insights, but, like, nothing, 02:29.000 --> 02:35.560 two important to actually make them, like, optimize their applications. So, there's, like, a 02:35.560 --> 02:40.360 sort of a problem we're trying to solve here, like, and then at my previous job, so we start to, 02:40.360 --> 02:44.360 to build, like, a stack or, like, computer energy and emissions monitoring stack, 02:44.760 --> 02:49.640 to sort of address these problems, right? I mean, so just to give you, like, a quick 02:49.640 --> 02:54.840 introduction, what it does is, like, we started this tool to, to estimate the energy consumption 02:54.840 --> 03:00.040 and the equivalent emissions of HPC, HPC, HPC bad jobs, because, like, our HPC cluster 03:00.040 --> 03:05.320 back then was, like, a very AI focused and, like, there's a lot of people, like, burning electricity 03:05.320 --> 03:09.480 by, like, finding, like, with the, with the fine jobs and stuff like this, and, like, people want 03:09.480 --> 03:13.560 to know how much of energy they're using it, because they have to report that, like, in the 03:13.560 --> 03:20.040 research papers, and thus, just one of the, thus, how the whole work started, like, and then we start, 03:20.040 --> 03:25.240 we, we noticed, like, the potential of the approach we took to develop this stack and we, 03:25.240 --> 03:30.280 you know, actually, extended the support to, but open stack and Kubernetes. So, the whole key, 03:30.280 --> 03:37.400 one, one stack of teams can, can monitor a HPC cluster based on SLAM, and open stack and 03:37.400 --> 03:42.840 Kubernetes, like, in a uniform way, so, giving, like, this resource manager, like, Nordstick flavor, 03:44.200 --> 03:47.800 an SSS system level style, it's not something that users have to use it, like, on the basis, 03:47.800 --> 03:53.000 like, something this is, it means an operators will instead, like, all on, on the entire HPC machine, 03:53.000 --> 03:58.120 or cloud or Kubernetes, and you're, like, get, like, the metrics from graph on a dashbox, 03:58.120 --> 04:04.360 right, I mean, and the hard of the teams, basically, the C groups, I'll just give you, like, 04:04.360 --> 04:08.760 quick introduction of what is C groups. Perf sub system, I think most of the HPC folks know 04:08.760 --> 04:13.480 what is Perf sub system, and EBPF is coming, well, I mean, I'll try to give you a very short 04:13.480 --> 04:16.840 direction of what is EBPF, it's, like, pretty hard topic in the cloud network landscape. 04:18.440 --> 04:22.760 And, well, I must have very pragmatic person, like, I don't like to reinvent the wheels, 04:22.760 --> 04:27.080 which happens quite often in the HPC community, it might not be popular opinion here, but that's 04:27.080 --> 04:33.880 how I see it. So, I try to reuse, like, as much as I can from the existing tools, and then 04:33.880 --> 04:39.720 only implement the missing parts to make things usable, right. And so, if for that, like, we, 04:39.720 --> 04:45.000 I use, like, a lot of CNC, open source components, the promises, like, database to store all the data, 04:45.000 --> 04:49.000 and the graph on, for the visualization, and, like, for the people who don't want to use, like, 04:49.000 --> 04:53.240 a graph on, and they can always use, like, a CLI tool, which is also available with this stack. 04:54.120 --> 04:58.120 Just, like, a quick introduction of how games works, like, okay, just to understand, 04:58.120 --> 05:02.120 for, like, everybody's on the same page, like, control groups is the way that Linux actually 05:04.120 --> 05:10.520 managed the resources, like, CPU memory, I will GPUs whatever, right, I mean. So, effectively, 05:10.520 --> 05:14.840 for Linux, it could slow me job, but open stack virtual machine, like, a Kubernetes part is a 05:14.840 --> 05:23.240 C group. So, if you can manage to monitor the C groups on a given Linux server, we can manage, 05:23.240 --> 05:29.000 like, anything, a slow job, and, like, open stack VM or Kubernetes ports. So, 05:29.000 --> 05:35.320 so, just to show you, like, the whole idea of the games is, we have, like, a central unit here, 05:35.320 --> 05:39.880 like, which is games, and we pull the matrix from different sources, like, for, for, for example, 05:39.880 --> 05:44.840 from the kernel C groups, the CPU and data, I'm usage, and also at the node level, the same thing, 05:45.560 --> 05:52.440 and we pull the matrix, when available on the on the bare metal, from the RAPL and KPMC, 05:52.440 --> 05:57.880 as well as IPMR, I'd fish, HWM on, like, I think, we managed to add, like, support, 05:57.880 --> 06:03.240 to most of the energy resources, we have on the, on the bare metal level. And also, we get, like, 06:03.240 --> 06:09.720 the per-favorites from the per-psip system. We use the CPU, we, EPPF, to get, like, I go on network 06:09.720 --> 06:15.880 matrix, I'll explain, I'll be clear, and for the GPUs, like, we take, like, whatever, like, 06:15.880 --> 06:21.320 NVIDIA, DCGM, AMD matrix device export, it gives us, like, GPU matrix, and, we put all this 06:21.320 --> 06:25.400 matrix into the Prometheus database, and we showed them to the, using graph on the dashboards. 06:26.520 --> 06:30.520 And, well, this is how it pretty much looks, like, on the computer server, we have, like, typically, 06:30.520 --> 06:34.360 like, a Teams exporter, a Prometheus exporter this running, and also, if you have GPUs, 06:34.360 --> 06:40.760 yeah, in video, AMD, you get, like, the GP exporters that are running, they push all the matrix 06:40.760 --> 06:46.120 to the Prometheus database. At the same time, like, we take the, the real time in the static 06:46.120 --> 06:53.720 emission factors, like, from, from, you know, like, a pool, the dynamic factors, as well, into 06:54.520 --> 06:59.720 the Prometheus database. But, from the C groups, we get, like, a very basic metadata of the job, 06:59.720 --> 07:05.480 like, a job ID, for example, in, in the case of HPC, we need, like, a bit more metadata of the job, 07:05.480 --> 07:10.840 to be able to actually show this, the matrix on the, on the graph on the dashboards well. So, 07:10.840 --> 07:15.240 we pull, like, that has told the metadata from, like, a resource manager, for example, from 07:15.400 --> 07:20.680 a, from a, from a, from a Kubernetes. And we put them into an API server, which is basically a 07:20.680 --> 07:27.720 simple SQL database. And also, at the same time, we push this aggregated matrix into the API server, 07:27.720 --> 07:33.800 because this way, we can get, like, the aggregated energy usage emissions pretty easily, because, 07:33.800 --> 07:38.280 like, Prometheus doesn't, like, making a queries, like, okay, how much of energy consumption 07:38.280 --> 07:42.440 the last six months. I mean, these are, like, pretty heavy matrix on the Prometheus, and, and, and, 07:42.440 --> 07:47.080 that takes a lot of CPU, by, by, by storing aggregated matrix in the SQL database, 07:47.080 --> 07:52.680 we can get this information much faster. And, uh, and finally, we have, like, a little component 07:52.680 --> 07:56.680 called load balancer, it's more of a proxy than a load balancer, which can actually, 07:57.720 --> 08:03.880 I say, an optional component, I mean, um, um, imposed the access control on, on, on, on your 08:03.880 --> 08:08.120 metrics, because Prometheus doesn't offer any sort of access control. So, that's, what does it mean, 08:09.080 --> 08:15.240 one user can actually, technically, able to look at the metrics of any other user's jobs, 08:15.240 --> 08:20.360 by default, with Prometheus, and this, this load balancer, what it gives is, like, okay, the user 08:20.360 --> 08:26.200 cannot, uh, look at the metrics of other people's job rather than his or her own job sector. 08:26.200 --> 08:30.360 And, that's what, like, the load balancer does here, and you have, like, the user facing 08:30.360 --> 08:35.240 side of the graphana that, that show, like, the metrics to the, to the end users. And, that's, 08:35.320 --> 08:38.360 that's pretty much it, like, the architecture, like, a very simplified version of it. 08:39.960 --> 08:42.760 And so, a lot of the features, like, as I already mentioned, like, yeah, I, 08:42.760 --> 08:46.440 probably, like, do both energy and emissions, performance metrics, I will, it, 08:46.440 --> 08:51.240 file system, agnostic level, so it doesn't matter if you use a list for, or BGFs, or Spectrum scale. 08:52.280 --> 08:55.880 All the metrics are, are, are scrapped, like, a, I will matrix a scrapped in a, in a, in a, in a, 08:55.880 --> 09:01.480 in a, in a uniform manner, and it works out of, out of the box. Uh, uh, supports, like, a different 09:01.480 --> 09:07.640 energy resources, like an audio bill, his W1, Kray, BMC, via IPM and Redfish, supports both NVIDIA and 09:07.640 --> 09:13.000 DGPUs, and real-term access to the, to the metrics, and, or no, access control to the 09:13.000 --> 09:18.440 premises database. So, what does, uh, the end users can get, like, you know, they can get, like, 09:19.240 --> 09:23.160 dashboards, like, this in the real time for, like, a job usage, for example, like CPU stage memory, 09:23.160 --> 09:28.040 and, like, the host power usage here. And, you also get, like, the performance metrics, like, 09:28.120 --> 09:32.920 CPU frequency, instructions per cycle, and software events, and there's more, actually, 09:32.920 --> 09:36.680 like, wouldn't fit all the, all of them in the single-screen short, but, yeah, this sort of 09:36.680 --> 09:40.120 the things that you can get, like, actually, as I said, in the real-time as the job is running. 09:42.200 --> 09:46.360 Uh, the same with the GPU, like, uh, whatever, like, because we were using NVIDIA GPUs, 09:46.360 --> 09:51.400 so, what, what, whatever, like, NVIDIA gives us, it's a black box, we just show, uh, take them and 09:51.480 --> 09:56.680 show it to the end users. Um, uh, for the IO access, I like, you know, 09:56.680 --> 10:00.360 you can get, like, the read-and-write bandwidths and requests and errors, like, from different 10:00.360 --> 10:06.360 file systems, uh, different types of files system, like scratch work, stuff like this, and we get, 10:06.360 --> 10:16.680 like, the networks, uh, metrics as well, um, but only for, like, the, uh, IPv4 and V6, uh, but, 10:16.680 --> 10:23.320 both, like, the, uh, ingrisoned bandwidths and the packets, and this is how, like, uh, CLI 10:23.320 --> 10:28.440 Clang2 looks like, you know, something like, uh, what ASAC does, but, like, some more enhanced 10:28.440 --> 10:36.600 information, including the performance metrics and the energy and emissions. And also, that's, like, 10:36.600 --> 10:40.520 what, what, what, what, I should kill now is, like, for the end users, like, uh, uh, and this is what, 10:40.520 --> 10:45.560 like, a quest for administrators or operators can see, like, you know, uh, the, uh, the energy usage of the 10:45.560 --> 10:52.120 entire, entire cluster, um, divided by the partition, and also, like, the entire, uh, 10:52.120 --> 10:56.120 overall usage of the cluster, like, uh, based on the CPU, how much of the memory, the, 10:56.120 --> 11:02.280 each partition is using and stuff like this, like, well, um, and for example, this is 11:02.280 --> 11:05.480 can be, this can be, like, uh, uh, one of the metrics, one of the dashboards, 11:05.480 --> 11:09.320 can be really useful for, like, the operators, because here, what you see is, like, uh, 11:09.320 --> 11:13.960 I mean, I, I, I, I, I kind of mask them, but it's a project, and they use it in the project, 11:13.960 --> 11:18.920 and, like, how many jobs the projects have, uh, uh, uh, uh, submitted in the last six months, 11:18.920 --> 11:23.720 or something like this, and, like, the average CPU GPU usage, probably usage and stuff like this, 11:24.600 --> 11:30.040 with, with, with a, with a dashboard like this, like, the operators can quickly identify the users, 11:30.040 --> 11:34.840 which are using, like, uh, who are using the cluster in a bad way, like, say, for example, 11:34.840 --> 11:38.520 here, you can see, like, the user has submitted, like, 14 and talk, 50,000 jobs, 11:39.400 --> 11:45.480 with, like, uh, GPU usage only at 50% of the CPU usage at 6% and it's pretty easy for the, 11:45.480 --> 11:50.200 for the operators to identify these users and contact them and try to help the users to optimize 11:50.200 --> 11:53.480 the workforce to better use the cluster in the end of the day. 11:56.040 --> 12:00.200 Uh, well, some of the metrics, I think, we've already talked about them, like, uh, well, 12:00.200 --> 12:04.440 the only thing has been to talk is, like, uh, there's some RDM metrics available as well, but they're, 12:04.440 --> 12:09.640 like, very, very low level metrics and, uh, if you don't know how RDM works, like, it's really 12:09.640 --> 12:13.800 possible to make, uh, sense of them. We're talking about, like, a few pairs of memory regions, 12:13.800 --> 12:22.040 protected domains, and stuff like this. Oh, just to, tell you, like, the last point, like, 12:22.040 --> 12:27.320 all the metrics are part C groups, which means this part slum job. So, on your note, if you have, 12:27.320 --> 12:31.480 like, one job running on your note, 100 jobs running on them on the, on the note, doesn't matter. 12:31.480 --> 12:36.040 You always get, like, your metrics part, part job, like, even if the jobs are sharing the same note. 12:40.680 --> 12:44.360 So, yeah, it's all nice, like, you know, you get, like, this metrics in the real time, 12:44.360 --> 12:48.600 it's fair enough. Yeah. Okay. I can see, okay, my code is not doing well. What is supposed to do? 12:49.240 --> 12:52.040 Well, then what is the, what is the solution for that? Right. I mean, 12:53.000 --> 12:59.480 typically this is done by, like, uh, by the, uh, profiling. So, well, when the code is not running, 12:59.480 --> 13:04.440 well, it's not doing the things that it's supposed to do well. Well, we have to profile the code to understand, 13:04.440 --> 13:08.600 okay, what's the problem, right? The bottlenecks and hotspots. And typically, this is then with the profiling. 13:09.320 --> 13:14.040 The get-emnistic profiling, what you do is, like, on your laptop or on a login note, uh, to record all the 13:14.040 --> 13:20.920 call stacks, and it's very, very, very, uh, compute intensive, because, like, the, your process has to do a lot of, 13:20.920 --> 13:25.880 which is between the user, user, Canon and the user space in the kernel space, and that's, that's quite, 13:26.200 --> 13:31.160 the, this transition is quite, quite costly. And of course, you cannot always recreate the 13:31.160 --> 13:35.320 problematic scenarios in the, in the, in the, in the production, because, like, a profiling, you 13:35.320 --> 13:39.640 go down your laptop on a single process. It's not, there's a profiling, you go down, like, a MPI, 13:39.640 --> 13:46.680 when you're running it, like, more than one note, like, same. And what is the solution for that? 13:46.680 --> 13:51.000 Countless profiling, this is some, this is the topic that's, uh, cloud native folks, 13:51.400 --> 13:58.200 being, I mean, this is something that Google has, has proposed, like, a more than 10, 10, 15 years ago, 13:58.200 --> 14:03.400 I guess, almost, no. And, but this is something that cloud native folks use quite a lot, and it's 14:03.400 --> 14:09.560 completely EBPF-based. No instrumentation needed at all, and, like, it's very, very, very low, 14:09.560 --> 14:14.040 lower hold, and it's always on in the production, right? And it works out of the box for the 14:14.040 --> 14:20.040 compile language, just for, like, C, C plus, first, fourth, and then go. Uh, well, some of the, uh, 14:21.160 --> 14:25.320 tools that we have, like, do companies profile, like, graphana splung data, like, there's a lot of open 14:25.320 --> 14:30.920 stuff. And, like, Kim's exporters supports graphana pyroscop, graphana pyroscop is, like, equivalent of 14:30.920 --> 14:35.160 prometheus to store the profiles, like, basically, the, the, the, it stores, like, a times 14:35.160 --> 14:40.120 it's database, and the graphana pyroscop, which stores the profiles. So, what we can get if the 14:40.120 --> 14:44.760 continuous profiling. So, that, okay, this is the architecture now, it's very much, like, the same, 14:44.760 --> 14:49.560 like, the one difference is, like, the exporter sends the profiles of your, of your user applications. 14:50.520 --> 14:55.400 And, like, you, from the graphana, you can see these profiles, and what, what we're talking about 14:55.400 --> 14:59.400 the profiles when, like, is the same as, like, the typical flame graphs that you see, when you 14:59.400 --> 15:04.120 when you profile your jobs, and what you see is, like, a real slum job that is on your HPC cluster. 15:05.160 --> 15:09.560 It's a C plus plus code, I guess. And, what you can get is this sort of flame graph, like, in the real 15:09.560 --> 15:14.600 time, and what you see is, like, the names of the functions and the times spent inside each function, 15:14.600 --> 15:19.080 right, I mean. And the only thing that you need to do with your code to get, this is actually, 15:20.360 --> 15:24.680 include the debug symbols, without the debug symbols, I won't know what is the name of the function, 15:24.680 --> 15:30.360 and the, including the debug symbols, it doesn't, doesn't hit any, give any performance drop, 15:30.360 --> 15:34.120 generally. And this is for the C plus plus code, I mean, it's for the, for the compiler languages, 15:34.120 --> 15:39.400 work pretty well. And it works for the Python as well, even though it has, like, two levels of 15:40.040 --> 15:45.320 stack and winding process. But, yeah, for example, this is the example from the Python, and, 15:47.080 --> 15:52.520 and it works pretty well, what you see is, like, the name of the file, and, like, the name of the function, 15:52.520 --> 15:57.000 inside the file, and as I said, like, this is on the production machines, like, you know, 15:57.000 --> 16:01.640 I mean, our HPC center now is using for more, like, more than one and a half years, 16:01.640 --> 16:07.000 and this is a pretty happy, like, have to have this graphs in the real time, and things are working pretty well. 16:07.880 --> 16:14.360 Just to show you the export over here, like, average of 16 nodes, like, you know, 16:14.360 --> 16:20.120 this, like, almost the CPU usage, like, less than 5.5% and, like, the memory is, like, around 100 megabytes. 16:22.520 --> 16:27.560 And we did some benchmarks, like, with the, with some computer intensity task, 16:27.560 --> 16:32.520 like, without export, with export, with export and continuous profiling. But, from what we see, 16:32.520 --> 16:34.760 it's like, there's no huge performance sheet. 16:37.400 --> 16:41.960 Some of the technical details is 100% on the go, except the BPF programs, which is coming from 16:41.960 --> 16:47.320 EVPS, which I've written in C. I'm not going to talk about the capability awareness, 16:47.320 --> 16:52.120 EVPF and I want to mix. Okay, just to give what is EVPF, EVPF is, like, a way to 16:52.120 --> 16:58.200 to customize the kernel at the runtime without having to use the kernel module. So, you can inject 16:58.280 --> 17:04.120 any piece of code inside any point in the kernel at the runtime. I mean, it's not going to crash 17:04.120 --> 17:11.160 your kernel, because EVPF programs have to pass a very fair stage. And that's how we, we can get, 17:11.160 --> 17:16.680 like, a lot of information about I1 network. Just some more details about, like, testing and 17:16.680 --> 17:20.680 see, like, the battery test, it's, like, a pretty good, you need to discover it, and, like, there's 17:20.760 --> 17:28.840 a lot of interintests as well running in the C. I. And also, the packaging, like, all sorts of 17:28.840 --> 17:34.120 options at a while, will both, like, binaries, pre-compiled binaries, like O.C. image, and hand chart 17:34.120 --> 17:39.880 is also available to be able to run the thing. The final remort, so it's like, yeah, you know, 17:39.880 --> 17:43.640 the teams provide, like, a very complex solution, like, very provides both the metrics, 17:43.640 --> 17:48.920 and also, like, tools to profile your course in the runtime. In the production cases, at the real time, 17:48.920 --> 17:56.120 like, that helps you to optimize your workforce, and, like, better use the HPC classes and the 17:56.120 --> 18:01.960 energy associated with them. Well, it's running on over HPC classes of the joins, as I said, 18:01.960 --> 18:05.160 like, for more than, like, 100, 100, 100, 100, 100 years, like, with the scrap frequency of 10 seconds, 18:05.160 --> 18:11.880 so the users get, like, the metrics for each 10 seconds, for the CPUs, such and stuff like this. 18:13.720 --> 18:17.960 Currently, I'm working to, to support the cloud VMs using spec power database, 18:18.040 --> 18:22.120 because, like, on the cloud missions, we don't have any energy resources. The sources that gives 18:22.120 --> 18:26.520 the energy usage, so, like, we need to do some sort of modeling, and that's the modeling, 18:26.520 --> 18:33.160 we're trying to do the spec power database. The demo instances is there, like, to play around, 18:33.160 --> 18:39.320 like, if you want to see what how it feels like. And finally, like, I use, like, a Greek 5,000, 18:39.320 --> 18:45.320 like, one of the French testbed to, to actually develop most of this, talking with our data, like, 18:45.960 --> 18:51.240 this wouldn't be possible, like, in big thanks to the team that's maintaining that. 18:52.120 --> 18:56.120 And that's it, like, you have all the resources here, if you have any questions, I'm happy with it. Thank you. 18:56.120 --> 19:08.280 Thank you. 19:08.280 --> 19:08.600 Yes. 19:08.600 --> 19:15.960 The question is about, like, there are any questions, I'm not going to be able to, 19:15.960 --> 19:24.200 I'm not going to be able to, but I'm going to be able to, but I'm going to be able to, 19:24.200 --> 19:31.400 these about, like, there are any solutions for the ARM architecture. Well, I mean, I'm 19:31.400 --> 19:35.400 relying on whatever the vendors provide, like, if the vendors provide, for example, like, 19:35.400 --> 19:40.040 doesn't matter, even if it's ARM, the BMC is always there. Every vendor is supposed to put, like, 19:40.040 --> 19:46.120 a BMC chip on the BMC with always measure the total power intake. So, we can get, like, the same 19:46.120 --> 19:50.680 information from the BMC because of both IPM and right-fisher supported, even for the ARM. 19:54.280 --> 20:03.960 Yes, Samora, who else, if you will try to sample, and you say that that was a great, 20:03.960 --> 20:09.400 you know, the sampling about the sample. So, for everybody, there's an absolute value. 20:09.400 --> 20:16.120 But, for example, I mean, more interesting, I mean, of providing it. So, how much can I increase 20:16.120 --> 20:22.600 the sampling that, or how much it's flexible to say, okay, for these part of the, 20:22.920 --> 20:31.160 let's say, like, hundreds of milliseconds, I want to have an idea about the flexibility in that. 20:31.160 --> 20:35.720 Okay, the question is about, like, if you can change, like, the sampling frequency for, like, a given job, 20:35.720 --> 20:43.160 or a given note. No, with the primitius, well, I mean, it's possible if you contact your 20:43.160 --> 20:50.440 cluster admin to change, like, the, yeah, I mean, if you can change for a given note and then 20:50.520 --> 20:55.560 launch your job on this note, yes. But, like, if you really want to profile as, as I show, 20:55.560 --> 21:01.080 like, the continuous profiling, it does with the hundreds of frequency, which gives you a pretty, 21:01.080 --> 21:06.040 pretty, pretty good, like, ideas, already, but like, it gives you a flame graph, not, like, time series. 21:06.760 --> 21:07.240 Yeah. 21:07.240 --> 21:23.240 I mean, like, as I said, like, we're using primitius, oh, 21:38.200 --> 21:47.000 I mean, the question is, like, if I understood correctly, if I, if we can integrate, like, the 21:47.000 --> 21:51.880 matrix coming from the external power meter into the stack, I mean, like, you can push the 21:51.880 --> 21:56.040 data to the primitius, right? I mean, if your external power meters have some sort of exposure, 21:57.480 --> 22:01.080 you can position the primitius and then you can integrate that into the matrix, yeah. 22:08.200 --> 22:19.880 The question is, like, if I have, like, any, I mean, I guess to, to, to integrate graphana 22:19.880 --> 22:25.800 tempo to get the traces, that would be nice. I actually tried with Jagger to, to, to, to the 22:25.800 --> 22:31.160 EPPF as well to give, like, an MPI, I would traces, it is possible, but it's just, like, the 22:31.160 --> 22:33.640 question of, some time and effort, yes. 22:45.880 --> 22:49.960 Yes, the question is, like, the power matrix work for the multiple jobs on the same note, yes, 22:49.960 --> 22:55.240 as I said, like, all the matrix are coming from, um, C groups. So, if I have to split the 22:55.240 --> 22:59.240 power between multiple jobs, I have to do some sort of modeling. Currently, the modeling is 22:59.320 --> 23:05.800 based on CPU time and the memory usage of the job, compared to the, the node, and that's how I split the 23:05.800 --> 23:06.600 power. 23:23.880 --> 23:28.680 The question is, like, if it can, I saw the number of jobs by the total number of time, 23:29.320 --> 23:33.240 totally yes. I mean, it's something that graphana does, I don't do anything, and I, graphana, 23:33.240 --> 23:37.720 had this option of being that yes. 24:00.200 --> 24:07.400 So, the question is, like, is there any support for the hardware based power 24:07.400 --> 24:11.080 meters, and can you give the example of what is a hardware based power meter? 24:30.200 --> 24:32.200 Mm-hmm. 24:43.720 --> 24:53.400 Fair enough, I mean, as I said, just before, like, I mean, if you can, if you can measure this power, 24:53.400 --> 24:57.640 and then push it to the, to the, to the, to the promutius, of course, you can use it inside, 24:57.720 --> 25:00.840 and you have, like, as I said, like, the BMC is all, all, all already supported. 25:03.800 --> 25:05.000 Yes, please.