WEBVTT 00:00.000 --> 00:10.800 Hello everyone and welcome. My name is Bart and I'm here together with Igor. We're both 00:10.800 --> 00:14.800 data science software engineers from all the other. And today we will be talking about 00:14.800 --> 00:22.400 the journey of building and the architecture as well of Bobus Dev 4.0 Alpha. So I'll 00:22.400 --> 00:26.560 also the lessons we learned along the way and what currently available in the alpha release 00:26.560 --> 00:33.440 and what to be expected in the final release. But before we can go into that what even is 00:33.440 --> 00:38.480 open-step. So open-step stands for open short term energy forecasting and with short term 00:38.480 --> 00:44.480 energy forecasting we really mean forecasting up to hours or days at. So for example here you 00:44.480 --> 00:51.120 have a time series or load of a point in this case in the grid. So historical measurements 00:51.120 --> 00:55.920 with the red dotted line being right now and then open-step allows you to predict the future 00:55.920 --> 01:01.600 basically based on these measurements but also all their predictors like weather forecast energy 01:01.600 --> 01:07.680 prices and profiles and this way open-step will generate well actually multiple forecasts. So 01:07.680 --> 01:13.040 not just a single forecast but also with an uncertainty bandwidth because predicting the future 01:13.040 --> 01:18.400 is basically impossible so you need to have some kind of uncertainty bandwidth to cover your risk. 01:19.920 --> 01:25.200 And a bit more in detail what is OPSF so it's not just a model that allows you to produce these 01:25.280 --> 01:33.920 forecast. Actually OPSF is a model agnostic and specifically it's a Python package so it's a 01:33.920 --> 01:39.440 framework that allows you to create these equality energy forecast because creating energy forecast 01:39.440 --> 01:44.960 is more than just having a model putting in data and getting a forecast. It actually contains 01:44.960 --> 01:50.800 many machine learning steps. For example start with data pre-processing such as input validation, 01:50.800 --> 01:55.200 feature engineering, they need to train models and if you want a great forecast you need to use 01:55.200 --> 02:00.160 that trained model. If evaluating forecast is also just as important as making them because 02:00.160 --> 02:03.760 you want to know if they are good so you need to evaluate them on measurements of the past. 02:04.560 --> 02:10.160 And finally sometimes you also need something like post-processing and OPSF provides pipelines 02:10.160 --> 02:16.960 or workflows to allow you to easily do all of these steps without all the complexity that's 02:16.960 --> 02:25.600 needed to do it. And I also need to tell a bit why do you need OPSF and you probably heard a lot 02:25.600 --> 02:31.440 of this today already including these pictures. I think these are even outdated because these are 02:31.440 --> 02:36.160 from half a year ago. I think the new ones are even more wet that what is basically shows is that 02:36.160 --> 02:43.280 our grid is full and so OPSF can be used for many things but I'm here now going to explain one of 02:43.360 --> 02:49.120 the use cases we have at all the other so congestion management. So as you can see here the grid is full 02:50.240 --> 02:54.240 so what it means is that customers cannot be added to the grid and while we are of course 02:54.240 --> 02:59.680 reinforcing the grid that takes time and it cannot really solve this waiting list we currently have. 02:59.680 --> 03:05.200 So schools and factories they cannot be connected to the grid right now which has quite a societal 03:05.200 --> 03:11.840 impact as well but something we can do is actually we can connect them but that results in overload 03:11.840 --> 03:17.760 but we can do something about it because we have OPSF so we can forecast them the load of 03:17.760 --> 03:24.000 on these points in the grid that now if these extra customers connected and that means that sometimes 03:24.000 --> 03:29.360 you will have overload so here on the left you'll see load over time and OPSF can then predict in this 03:29.360 --> 03:34.480 case two days ahead there will be a moment of a few hours where the load will exceed the limit 03:34.480 --> 03:39.680 of the equipment that's at that point. What we then can do because we know it in advance we can 03:39.680 --> 03:44.960 call a customer and ask them to basically reduce their energy consumption or generation 03:46.560 --> 03:53.200 against some kind of conversation so in the end we prevent this peak moment from happening because 03:53.200 --> 03:59.360 the customer reduces their energy activity basically and so this is the congestion management process 03:59.360 --> 04:03.760 and so you really need these accurate forecasts of these peak moments so that's where we use OPSF 04:04.400 --> 04:10.480 but actually over the years more and more use cases are host. For example transport forecast 04:10.480 --> 04:15.280 grid loss forecasts with different things as a district eating which has nothing to do with 04:15.280 --> 04:20.960 electricity at all so that's one of the use cases from the community so not not even 04:20.960 --> 04:27.440 all the other and so as more and more use cases came to be for the short time energy forecasting 04:27.440 --> 04:32.960 we were at some point where we wanted to further improve the flexibility of the current OPSF 04:32.960 --> 04:40.160 to really allow these more these newer use cases to be well even better supported than they're 04:40.160 --> 04:45.040 currently are so this brought us to the redesign of OPSF we were talking about today 04:45.840 --> 04:50.560 so we had some goals for this so we really want to keep focusing on it having to be a 04:51.520 --> 04:56.560 machine learning library so for creating forecast and for the evaluation while also providing these 04:57.360 --> 05:05.760 presets for common use cases like I mentioned and then OPSF also targets different target audiences 05:06.400 --> 05:12.560 so for example the researchers and experimentation phase so those are mostly working with in 05:12.560 --> 05:19.440 Jupiter notebooks with data sets of mostly focusing on metrics and and floating functionality 05:19.440 --> 05:24.800 so that's already well supported within the current OPSF that we've been OPSF before we want to 05:24.800 --> 05:32.560 improve this by also making the back testing framework more realistic second use user group for 05:32.560 --> 05:36.800 OPSF would be small skill deployment so let's say our single data science team 05:37.760 --> 05:43.280 and you need to create forecast for your company so from end to end so from retrieving measurements 05:43.280 --> 05:49.040 and forecast a rather forecast that do storing the forecast made by OPSF so by providing 05:49.040 --> 05:54.560 reference implementations or example implementations we want to support those use cases 05:55.200 --> 06:00.000 which is already one of the strong points of the current OPSF version so we want to keep doing that 06:00.640 --> 06:06.880 with something that's really new with OPSF before one of the main goals of this new redesign 06:06.880 --> 06:12.800 is to target the enterprise integration so in larger companies like we currently now are as 06:12.800 --> 06:17.520 as early on there we have quite a complex software landscape so many custom APIs but also there 06:17.520 --> 06:25.120 can be policies on what OPSF where you can or cannot use and by having OPSF flexible or even 06:25.120 --> 06:32.480 more flexible and modular the goal is to allow OPSF to be really easy integratable in search systems 06:34.640 --> 06:39.280 so that are basically the goals of this major redesign and so that brings quite some challenges as 06:39.280 --> 06:45.200 well so you can imagine so to have this flexibility and these different use cases to support 06:45.200 --> 06:50.960 them you really need an opinionate of design so not from a single use case such as congestion management 06:50.960 --> 06:57.760 you but you really need to be really an opinionate in that something else we found is that 06:58.480 --> 07:03.200 data availability constraints is something that also happens quite a bit so for example measurements 07:03.200 --> 07:07.120 can come in delayed or rather forecast that is used in the forecasting process they can have 07:08.400 --> 07:14.640 delays so I think native support within OPSF for this is really huge when for complexity 07:14.640 --> 07:19.360 but also for back testing situations where you can then take into account these data delays 07:19.360 --> 07:22.880 so that's also one of the key points both the redesign that we're working on 07:24.000 --> 07:29.760 flexibility kind of makes sense if you want to be more flexible and target also enterprise integration 07:31.360 --> 07:37.040 and all of this of course we've out compromises on performance so not just the model forecasting 07:37.040 --> 07:42.640 quality but also the performance of running OPSF itself in case you need to scale up and when 07:42.640 --> 07:48.800 multiple forecast a day and all of this of course you do not want to reinvent the wheel because 07:48.800 --> 07:53.280 well this is quite a huge redesign otherwise you will be there forever and they're very 07:53.280 --> 08:00.080 great projects out there that we can look into so that's also where we started so with design 08:00.080 --> 08:04.160 and as this OPSF is really a community project we did this together with the community 08:04.160 --> 08:09.760 that many meetings planning sessions brainstorms on what are the current pain points strong 08:10.080 --> 08:14.880 back and we improved what do we need for a proper redesign and in the end we had like this 08:14.880 --> 08:20.080 detailed design proposal that we shared with the community had many feedback runs on and for 08:20.080 --> 08:23.440 for that we used slack and confidence there's a great great places to 08:24.480 --> 08:28.240 well get feedback from the community as well just like the community meetings 08:30.480 --> 08:34.480 and in the end well quite a nice design but part of it was also looking into 08:34.480 --> 08:45.360 existing projects right because we do not want to reinvent the wheel so that's something 08:45.360 --> 08:53.760 I will pass over to Egor thank you so yeah when designing the project we wanted to not only 08:53.760 --> 08:59.680 look at the community but also at what is out there and investigate existing forecasting 08:59.680 --> 09:07.440 libraries and how they fare on load forecasting and yeah or use cases and to kind of invent 09:07.440 --> 09:19.600 advice what's our different points but what do they do well what what do they miss what kind of gap 09:19.600 --> 09:28.560 can open stuff fill in this space so we get it best practices things that we shoot copy and we 09:28.560 --> 09:35.680 showed not reinvent ourselves but also gather things that yeah all every needs to do well 09:35.680 --> 09:43.360 to be an addition for the energy forecasting space and taking all of this into consideration this 09:43.360 --> 09:52.240 is the architecture we have come up with it contains of multiple modules which are mostly self 09:52.240 --> 09:58.800 contained and yeah are really useful to keep the frame up modular because some teams may do 09:58.800 --> 10:04.720 forecasting but some teams may be focused on evaluation or actually only consuming the forecasts 10:05.680 --> 10:10.880 that's bring this me to the first core component of our library which is core which contains all the 10:10.880 --> 10:17.520 data types interfaces and everything that is needed for what comes after this way everyone has 10:18.480 --> 10:25.120 a defined set of interfaces they can use to extend open stuff similarly yeah we multiply 10:25.120 --> 10:32.960 exceptions and testing utilities for the next libraries and then this is all used by the libraries such 10:32.960 --> 10:41.440 as open stuff models which is really the core of open stuff it has the forecasting models but also 10:41.440 --> 10:47.840 the data processing pipelines and the main specific transformations we need for energy forecasting 10:48.560 --> 10:55.600 similarly we decided to to focus really on expandability and have a way to explain the actual 10:55.600 --> 11:05.200 forecast you're getting and yeah presets so people can start using it quickly and audio modules 11:05.200 --> 11:13.280 include major learning which is based on a much stronger way of forecasting using modern models 11:13.280 --> 11:20.000 and assembling it and then we have open stuff beam which stands for a back testing evaluation 11:20.000 --> 11:26.720 analysis and metrics and this is a yeah very important library for us it's then from an internal 11:26.720 --> 11:32.720 project and to basically answer a question are the changes that I'm making in the model are 11:32.720 --> 11:39.440 they are significant and do the Gaussian improvements are regression that's a way have it there and 11:39.440 --> 11:45.520 we have made it a core part of open stuff and finally a foundation of models which we want to 11:45.520 --> 11:50.640 talk too much about because it's still a broken progress but the main idea is to have pre-trained 11:50.640 --> 12:01.600 models that you can use for forecasting your energy data so once we got our design ready and some 12:01.600 --> 12:08.000 many show drafts on how we want to do this we started with a public back look in our 12:08.000 --> 12:13.920 open stuff get a project we have created a bunch of stories and tasks that we wanted to do to get 12:14.560 --> 12:23.200 open stuff for ready we defined a few milestones and and this is in this way yeah people can follow 12:23.680 --> 12:32.240 us as how we build the library and yeah see where we are at and what is planned similarly we have 12:33.120 --> 12:38.720 focused on a few more tasks that are mostly self-contained a small to market them as good for 12:38.720 --> 12:45.440 stations and pre-refinement to get new contributor started these tasks currently limited in context 12:45.440 --> 12:54.400 and yeah really nice for people wanting to get into the space and to on top of that we are 12:54.400 --> 13:01.200 host co-coding sessions every eight weeks where we and our community members come together and 13:01.200 --> 13:07.520 work together on tasks it could be designed but it could be also the tasks that such as good 13:07.600 --> 13:16.480 first issues and things that you want to implement it or brainstorming. Support the first iteration 13:16.480 --> 13:22.160 so once we got our design back look in the natural next thing is to start developing but 13:23.120 --> 13:31.120 as part of this development process we wanted to get a really good setup on to make it easy for 13:31.120 --> 13:40.000 users to contribute but also to quickly iterate on for building a better open stuff. A few tools that 13:40.000 --> 13:46.160 we use with that is for type safety which really helps us with refactors and changing takes we get 13:46.160 --> 13:55.760 immediate feedback on what you're doing similarly documentation tests. Similarly the commutation tests 13:56.480 --> 14:02.640 which well we're building we're also waiting the commutation and some plans change some 14:03.600 --> 14:10.400 that's way by incorporating it as part of our unit tests we are also keeping the computation up to date 14:11.040 --> 14:17.680 which is up a good feedback for us and then you fear of SIGCD which are really useful 14:18.480 --> 14:24.240 development tools we have nowadays really fast and really support our Python development 14:25.200 --> 14:32.240 finally what I wanted to talk about a lot what I wanted to talk about is the benchmarking which is 14:32.240 --> 14:42.080 really a big part of our regression testing strategy. Our benchmarking as I said earlier 14:42.080 --> 14:47.520 is part of open stuff beam which is a package that's used for evaluating forecasts but to 14:47.520 --> 14:54.160 validate the actual forecast we also need data and that's why we have open source Leander 2021 14:54.160 --> 15:00.560 energy forecasting benchmark which includes 50 plus different energy signals ranging from 15:01.840 --> 15:10.000 solar parks, wind parks, transformers, MP feeders and much more. It gives us a good sample on 15:10.000 --> 15:16.640 different use cases for energy forecasting and allows us to test our models really thoroughly 15:18.480 --> 15:26.560 and this is yeah really out of the box Renable you can clone the repo and do the test yourself 15:28.640 --> 15:34.400 what we also did is we started using open stuff really early in our development we in our team 15:34.400 --> 15:41.600 also really focused on improving our forecast for congestion management and by using really early 15:41.600 --> 15:48.000 version of open stuff we were able to get early feedback see what we're doing is the writing 15:48.000 --> 16:01.280 and refine but also find bricks early. So where are we at now we have achieved feature parity with 16:01.280 --> 16:08.960 open stuff E3 which was the major milestone for us and which has become our alpha release which contains 16:09.040 --> 16:14.960 the full forecasting module and evaluation framework. Overall in this case means that 16:15.920 --> 16:20.960 we still reserve some rights to make some non-trivial changes but we are very confident 16:21.680 --> 16:26.720 in the in its current capabilities and in fact we are already using it at the Leander 16:26.720 --> 16:34.320 in production. In our team we make more than 10 for more than 10,000 different locations across 16:34.320 --> 16:44.240 the grid and it's yeah steadily growing. So what are the lessons learnt or yeah from this 16:44.240 --> 16:52.560 development until now one thing is make it really safe to experiment yeah by creating the regression 16:52.560 --> 16:59.840 testing it makes really easy for our researchers to try different ideas see what works see what 16:59.840 --> 17:06.640 doesn't and get immediate feedback because yeah energy forecasting complex and sometimes we need to 17:06.640 --> 17:15.760 get a lot of data to get started but also also you may also need to wait a few days for the data 17:15.760 --> 17:21.360 or the forecast to come in by creating our benchmark for simulating it we are festric in this process 17:22.160 --> 17:31.040 in fact there's been also a yeah an experiment of ours to see if you could improve this process 17:33.440 --> 17:39.040 yeah prepare for the contributors we have seen quite some good contributions for community 17:39.040 --> 17:47.680 into open stuff and open stuff for and yeah it's really helps people to get started early and 17:47.840 --> 17:55.680 look to look at some issues that are really good ways to get into the forecasting design for 17:55.680 --> 18:01.280 my ability from the one we found that really useful because our library do quite a lot of things 18:01.280 --> 18:08.080 because forecasting is the very info of process and this way it allowed us to focus and structure 18:08.080 --> 18:15.520 our code in a way that yeah could be the early test it and yeah is something as achievable 18:15.520 --> 18:22.000 and finally real world testing we need to know what are you building for and actually use it for 18:22.000 --> 18:35.360 it to see if it works or next steps for our library we are currently in alpha release and we are 18:35.360 --> 18:40.960 yeah steadily building towards a stable release it's currently a bit it's working and it 18:40.960 --> 18:46.160 but it's rough around the edges for the documentation so our goal is to get really the 18:47.120 --> 18:53.600 well documented website started with good getting their started guides for both researches 18:53.600 --> 19:00.080 but also for people who want to run it in production and work with it which brings 19:00.080 --> 19:06.960 it to the next point development deployment guides and deployment examples yeah and my 19:06.960 --> 19:15.840 lobes is quite complicated thing and if you can get some examples for users to deploy it helps 19:15.840 --> 19:24.320 of a lot and finally the new things made to learning module is steady on its way and yeah we are 19:24.400 --> 19:33.840 clicked really work on it and for days you know models is also yeah on its way you bring me to the 19:33.840 --> 19:41.120 final slides yeah thanks Igor so if you before you go into the questions because this was 19:41.120 --> 19:45.520 our presentation that if you want to know more about it over staff we have quite some 19:46.480 --> 19:51.360 quite some links to get to start it or have been more information so we have like a slack for 19:51.440 --> 19:56.640 communication to the teams for to the community we have an energy project page to get to 19:56.640 --> 20:02.400 pray it's also the hooking phase data set that that Igor mentioned so if you want to actually 20:02.400 --> 20:09.520 start using OBSF but you don't have data but still want to see what happens we have this this 20:09.520 --> 20:14.960 public benchmark data set we've already screwed within OBSF to run it so that you can really 20:14.960 --> 20:20.880 experiment for yourself and see what it does and finally I've to mention we we have these nice 20:20.880 --> 20:24.560 we have a really nice community so we have these for weekly community meetings as well 20:25.520 --> 20:29.840 including coding sessions I think like every eight weeks or so where we sit together with the 20:29.840 --> 20:35.200 community and think about what our features we want to add and then do it together so if you 20:36.160 --> 20:42.560 find that interesting it's all open to join so here are the links to getting contact with us 20:42.560 --> 20:46.160 and with that thank you for listening and let's go to the questions 20:56.880 --> 20:58.320 yes 20:58.320 --> 21:04.160 I'm not a user of open steps so it makes very much a big question but does it involve some 21:04.160 --> 21:10.480 read topology or do you use a need the times three on nodes that may be aware of 21:11.440 --> 21:16.160 yes so the question was do those OBSF use grid topology is that required 21:17.440 --> 21:22.000 OBSF currently works is it's really point point based so let's say you have 21:23.200 --> 21:26.560 you have for example medium voltage routing onto forecast each of those points 21:27.600 --> 21:31.760 the ideas that you have measurements on each of those points and then forecast them but if you 21:31.760 --> 21:38.080 want to include grid topology as well you can also use the power grid model in combination 21:38.080 --> 21:42.880 I think there's also a paper published already about this about this use case where you can 21:42.880 --> 21:49.680 use both the topology in combination with OBSF to even further improve forecast quality so you can use both basically 21:50.560 --> 21:54.560 yes 21:59.200 --> 22:01.200 sorry can you repeat 22:05.920 --> 22:12.560 yeah do we have other DSOs that are involved within OBSF so we have RTE and RTE international so they're 22:12.560 --> 22:23.440 not DSOs there but like DSOs themselves not yet so it's OBSF really as the DSO but we also have 22:23.440 --> 22:29.360 for example Siegel in Sweden we were really involved as well but they are mostly like a 22:29.360 --> 22:36.240 consultancy helping local DSOs or even for district eating but as for the DSO party where 22:36.320 --> 22:48.480 that's only under the only one's currently is a question is what accuracy do we achieve with the 22:48.480 --> 22:55.440 forecasting that really depends on what you're forecasting so it's a tough question to answer 22:56.640 --> 23:01.920 because it also depends on what your use case is because if your use case is congestion management 23:01.920 --> 23:07.120 forecasting then you really only interest it in those peak moments so what then how which of 23:07.120 --> 23:12.000 forecasting quality is during nighttime doesn't really matter so then you want to use different 23:12.000 --> 23:19.600 metrics so then like RME or those types of accuracy metrics make less sense so it depends on your 23:19.600 --> 23:25.760 own your use case and I would say it's pretty good but maybe it's best to see for yourself if you 23:25.760 --> 23:33.200 also run the benchmark with the working phase data set they can with with OBSF being you get 23:33.200 --> 23:36.960 all these different kind of plots for different kinds of metrics so they can really get an in-depth 23:38.560 --> 23:44.560 insight into how well these these forecasts perform but of course it also it's not just also the 23:44.560 --> 23:50.000 models it's also like the input data that you choose so if you are input data is it's a bad quality 23:50.000 --> 23:54.400 then your forecast will also be so it's always a different difficult question to to answer 23:57.440 --> 24:06.320 yes how far ahead did you consider short-term we would consider well up to seven days because when 24:06.320 --> 24:14.000 you extended 37 days as your weather forecast either will not be available in for example 15 minute 24:14.000 --> 24:20.320 resolution which is typical but what is used within this electricity grid it also here energy forecast 24:20.320 --> 24:24.640 or your weather forecast will be of such that quality that you cannot really foresee a solar peak 24:24.640 --> 24:30.640 anymore because you will not know if it will be cloudy or not so that's the when we talk about short-term 24:30.640 --> 24:39.600 energy forecasting it it is hours until days with a max of around seven days yes 24:44.320 --> 25:04.480 yeah so the question is what makes open stuff specific or like unique compared to general 25:05.200 --> 25:18.720 forecasting is that during this and quickly yeah okay what makes it specific 25:18.720 --> 25:25.200 open stuff for for energy forecasting compared to all the types of forecasting so within open stuff 25:25.200 --> 25:31.440 there's also a lot of domain knowledge included in for example within the feature engineering part 25:31.440 --> 25:35.760 which is quite important so for example when you provide a weather forecast 25:37.760 --> 25:45.600 open stuff will if you want it can automatically derive for example based on solar radiation 25:45.600 --> 25:52.240 and the the temperature if you weather forecast what the typical pp panel would generate and these 25:52.240 --> 25:59.760 type of feature engineering will you make it get the edge on forecasting because most of the 25:59.760 --> 26:03.840 currently the models within the open stuff they are based on classical machine learning so 26:03.840 --> 26:09.920 we're relatively simple models like decision trees linear regression so that's really what gives 26:09.920 --> 26:15.280 it the edge of course when you add the the the the deep learning things will change but that's 26:15.280 --> 26:20.000 for for another day thank you for your questions