WEBVTT 00:00.000 --> 00:07.000 You hear me? 00:07.000 --> 00:08.000 All right. 00:08.000 --> 00:10.000 So, my name is Ricardo Delphin. 00:10.000 --> 00:12.000 I will be giving me a talk today on production 00:12.000 --> 00:15.000 and anything wrong when you have no choice with basal. 00:15.000 --> 00:18.000 Before I get started, just a little bit of background on myself. 00:18.000 --> 00:21.000 Mainly work previously as a production engineer 00:21.000 --> 00:24.000 and a variety of fields, starting in servers 00:24.000 --> 00:27.000 and moving on to robots, be it autonomous vehicles 00:27.000 --> 00:29.000 or at the moment humanoid robots. 00:29.000 --> 00:32.000 So, a lot of my work has been around. 00:32.000 --> 00:34.000 Has been dedicated around production, 00:34.000 --> 00:37.000 izing systems, getting them working on real deployments 00:37.000 --> 00:40.000 and getting them out in the case of vehicles on the road 00:40.000 --> 00:42.000 to actually turn them into real products. 00:42.000 --> 00:45.000 So, with that in mind, I've got to start it. 00:45.000 --> 00:47.000 So, why do you pick Ross, right? 00:47.000 --> 00:50.000 I assume given the amount of talks on Ross who've had here 00:50.000 --> 00:52.000 most people here, you have used the Ross, 00:52.000 --> 00:54.000 you were familiar with it. 00:54.000 --> 00:56.000 And one of the bigger reasons why people lean towards Ross 00:57.000 --> 00:59.000 and the next perfect sense is it's a very standard tool 00:59.000 --> 01:01.000 to use in robotics research. 01:01.000 --> 01:02.000 It's very easy to set up. 01:02.000 --> 01:04.000 You can just compose things, you add more nodes, 01:04.000 --> 01:06.000 you add more things you're launching and things 01:06.000 --> 01:08.000 just kind of work out of the box, right? 01:08.000 --> 01:12.000 It's a very valid reason to use robotics framework like this. 01:12.000 --> 01:15.000 However, it does have a few issues, a few caveats 01:15.000 --> 01:17.000 that can cause issues when you're going into production, 01:17.000 --> 01:20.000 especially some of the big issues you sometimes see. 01:20.000 --> 01:22.000 I mean, this is a personal gripe, 01:22.000 --> 01:25.000 but the package is the primary deployment mechanism 01:26.000 --> 01:28.000 I think can be a bit of an issue. 01:28.000 --> 01:30.000 You're tied to specific Ubuntu releases. 01:30.000 --> 01:32.000 There's of course ways around this, 01:32.000 --> 01:35.000 but as a core, this is kind of what you're centered on. 01:35.000 --> 01:38.000 There's, you know, I think we've all found 01:38.000 --> 01:41.000 non-deterministic bugs at some point in Ross. 01:41.000 --> 01:44.000 You know, I've, I've myself encountered multiple times 01:44.000 --> 01:47.000 issues with shared memory topics 01:47.000 --> 01:50.000 and when you're working with different kind of components. 01:50.000 --> 01:52.000 You know, a lot of this can cause issues 01:52.000 --> 01:54.000 once you're actually deploying kind of a large set of systems, right? 01:54.000 --> 01:58.000 Now, unfortunately, sometimes you don't have a choice, right? 01:58.000 --> 01:59.000 Like, you come into a company, 01:59.000 --> 02:00.000 there is a system. 02:00.000 --> 02:03.000 It's been built in oftentimes 02:03.000 --> 02:04.000 rebuilding a thing from scratch. 02:04.000 --> 02:05.000 It's not a good idea, right? 02:05.000 --> 02:08.000 It's just not a reasonable thing to do 02:08.000 --> 02:09.000 and can be a waste of time. 02:09.000 --> 02:11.000 So what, what you want is you want to find a way 02:11.000 --> 02:13.000 of working with a system you've got, right? 02:13.000 --> 02:15.000 So some of the things that I generally recommend 02:15.000 --> 02:18.000 when you're looking at how to kind of improve a system like this 02:18.000 --> 02:22.000 is for something that you want to have clear dependency definitions. 02:22.000 --> 02:25.000 You want to have some form of build isolation, right? 02:25.000 --> 02:27.000 Something that actually gives you her medic builds, 02:27.000 --> 02:29.000 something that gives you portability and makes it easy 02:29.000 --> 02:31.000 for you to move your system from one thing to another. 02:31.000 --> 02:33.000 And to add new architectures as you go, right? 02:33.000 --> 02:36.000 And the build isolation can be really important. 02:36.000 --> 02:38.000 And this is where basal really comes in 02:38.000 --> 02:40.000 and shines quite well, right? 02:40.000 --> 02:42.000 So you can absolutely use basal with Ross. 02:42.000 --> 02:45.000 I will be basing my examples off of this 02:45.000 --> 02:46.000 good hub repo that I made. 02:46.000 --> 02:47.000 Just a couple small examples. 02:47.000 --> 02:49.000 You're welcome to look at the, 02:49.000 --> 02:52.000 pause them page to, you know, 02:52.000 --> 02:54.000 this lies in everything are there. 02:54.000 --> 02:57.000 So you can absolutely use basal to build Ross. 02:57.000 --> 03:00.000 My recommendation at the moment is to use this set of rules 03:00.000 --> 03:02.000 by Melan Vukov, fantastic. 03:02.000 --> 03:03.000 They work really well. 03:03.000 --> 03:05.000 There's other ones like Open, 03:05.000 --> 03:07.000 the apex AI has their own set to do 03:07.000 --> 03:09.000 very similar set of things and a bunch of other ones. 03:09.000 --> 03:12.000 But the key thing here is that this lets you do things like building 03:12.000 --> 03:16.000 Ross from source and have no working launch files 03:16.000 --> 03:18.000 and a bunch of other features, right? 03:18.000 --> 03:21.000 It makes your build independent from your Ross list, 03:21.000 --> 03:23.000 it limits this to you because you're just building from source. 03:23.000 --> 03:25.000 And you can do other things then with basal 03:25.000 --> 03:27.000 to improve permitticity, right? 03:27.000 --> 03:31.000 It also kind of limits this set of actual system dependencies you have. 03:31.000 --> 03:33.000 So just to show you what that actually looks like 03:33.000 --> 03:34.000 when you start writing code. 03:34.000 --> 03:36.000 As you can see here on the left, 03:36.000 --> 03:37.000 this is your basal rules. 03:37.000 --> 03:39.000 So you basically just define a binary, 03:39.000 --> 03:42.000 put in your dependencies, our CLCPP is there. 03:42.000 --> 03:45.000 You add a, you can do similar thing with Python. 03:45.000 --> 03:47.000 You can define Ross launch files 03:47.000 --> 03:50.000 and you can see a Ross launch part here on the right. 03:50.000 --> 03:52.000 It looks very similar to what you usually expect. 03:52.000 --> 03:55.000 The only difference is this executable name. 03:55.000 --> 03:57.000 The repo has some examples and how to do that 03:57.000 --> 04:00.000 or the rules repo has some examples. 04:00.000 --> 04:02.000 I do have a video if you want to look at this on line, 04:02.000 --> 04:04.000 but you can essentially just do basal run, 04:04.000 --> 04:06.000 start the node, start a launch, 04:06.000 --> 04:08.000 and just have it running. 04:08.000 --> 04:10.000 And it's quite quick to set up 04:10.000 --> 04:12.000 and it gives you a good baseline to work with, right? 04:12.000 --> 04:15.000 But this doesn't give you immediate hermitticity, right? 04:15.000 --> 04:17.000 And it doesn't really isolate your environment. 04:17.000 --> 04:19.000 It doesn't isolate your run environment entirely. 04:19.000 --> 04:21.000 It just gives you a base to build things on. 04:21.000 --> 04:23.000 So once you're actually deploying, 04:23.000 --> 04:24.000 you've got a lot of options, 04:24.000 --> 04:26.000 and the simplest is just using containers. 04:26.000 --> 04:28.000 This is what I will recommend. 04:28.000 --> 04:30.000 Most people do as their first instance. 04:30.000 --> 04:31.000 Just if you have nothing, 04:31.000 --> 04:33.000 just deploy on a container. 04:33.000 --> 04:35.000 Because this will take you from having, 04:35.000 --> 04:38.000 you know, running on your bare system to just 04:38.000 --> 04:40.000 running in something that is isolated 04:40.000 --> 04:43.000 and where you decide what devices you're actually talking to 04:43.000 --> 04:45.000 and what you have access to, right? 04:45.000 --> 04:47.000 It's very easy to set up. 04:47.000 --> 04:51.000 And the nice thing is you can also just make it run at Boots, right? 04:51.000 --> 04:53.000 One of the common issues with the loss 04:53.000 --> 04:55.000 when you're deploying into real systems is you 04:55.000 --> 04:58.000 need to find a way of making it run from Boots. 04:58.000 --> 05:00.000 And just the easiest way is you can get a container 05:00.000 --> 05:02.000 and put a system to service around it. 05:02.000 --> 05:04.000 I recommend this two set of rules, 05:04.000 --> 05:08.000 rules of CI for building a Docker containers in basal, 05:08.000 --> 05:10.000 and rules of this tool list for using the 05:10.000 --> 05:13.000 Docker containers when setting these up, right? 05:13.000 --> 05:14.000 So you can see right here, 05:14.000 --> 05:16.000 kind of example, what that looks like. 05:16.000 --> 05:17.000 You define a tarball. 05:17.000 --> 05:19.000 You include all the run files. 05:19.000 --> 05:21.000 So the run files here being, 05:21.000 --> 05:25.000 both the binary that you're running and all the dependencies 05:25.000 --> 05:26.000 that come down from underneath it. 05:26.000 --> 05:28.000 So this will generate a tarball. 05:28.000 --> 05:30.000 You just slap them on an OCI image. 05:30.000 --> 05:32.000 I based it off of, 05:32.000 --> 05:35.000 it's basically just a bunch of 20204 image 05:35.000 --> 05:38.000 with a couple of system libraries installed on top. 05:38.000 --> 05:41.000 And then you just have this OCI load thing. 05:41.000 --> 05:45.000 That just lets you load it into your Docker environment, right? 05:45.000 --> 05:47.000 And again, feel free to look at this video offline, 05:47.000 --> 05:50.000 but you can just do basal run that target 05:50.000 --> 05:54.000 and suddenly you've got a Docker container that just works, right? 05:54.000 --> 05:56.000 Another option you have is the point things 05:56.000 --> 05:57.000 by a packages, right? 05:57.000 --> 06:01.000 So this is going kind of back to WMPackage to a degree 06:01.000 --> 06:04.000 but separating it out from your distro. 06:04.000 --> 06:08.000 Basal also has in rules package a bunch of things 06:08.000 --> 06:11.000 for building the deviant packages. 06:11.000 --> 06:14.000 And much like before, you can just figure out a way 06:14.000 --> 06:17.000 of doing system deployment by setting up system D service, right? 06:17.000 --> 06:19.000 And so that's what you're seeing here. 06:19.000 --> 06:22.000 You basically have your same package tarball 06:22.000 --> 06:24.000 with all your dependencies here. 06:24.000 --> 06:25.000 You set up a system D service, 06:25.000 --> 06:27.000 like the one here on the right. 06:27.000 --> 06:31.000 And then you just define a package, a deviant package, right? 06:31.000 --> 06:33.000 And it'll tarball everything up. 06:33.000 --> 06:36.000 You can even define the post install and pre-remove scripts 06:36.000 --> 06:40.000 so that you can set up the system D service to start and run. 06:40.000 --> 06:42.000 And that basically just works, right? 06:42.000 --> 06:45.000 So this is again something that I really recommend you try out 06:45.000 --> 06:48.000 if you haven't really done a lot of work deploying 06:48.000 --> 06:50.000 raw-son systems and you're trying to figure out 06:50.000 --> 06:54.000 a productionizer here, a robotic system. 06:54.000 --> 06:56.000 Another option you have is deploying with images. 06:56.000 --> 06:59.000 And I think this is kind of a very common next step 06:59.000 --> 07:00.000 that you go, right? 07:00.000 --> 07:05.000 So at some point, you will encounter the issue that your drivers 07:05.000 --> 07:09.000 on your robot are kind of different across different robots, right? 07:09.000 --> 07:12.000 Like someone maybe has some scripts that they run on the robot 07:12.000 --> 07:16.000 to install some random kernel packages that install 07:16.000 --> 07:19.000 the camera drivers for DMSL or whatever. 07:19.000 --> 07:21.000 And someone forgot to install them on the next one 07:21.000 --> 07:23.000 or they have a different version. 07:23.000 --> 07:26.000 A very common recommendation here is instead of trying to do that, 07:26.000 --> 07:30.000 manually try to check everything into your repo, 07:30.000 --> 07:33.000 set it up as an image that you generate, 07:33.000 --> 07:35.000 and then flash that image. 07:35.000 --> 07:37.000 That way you kind of have your own custom, 07:37.000 --> 07:39.000 almost cost them operating system, right? 07:39.000 --> 07:41.000 Yachto is a very common way of doing this. 07:41.000 --> 07:45.000 So it's definitely one of the ways I've seen it done the past. 07:45.000 --> 07:47.000 And what you can do is you can build this Yachto base image, 07:47.000 --> 07:52.000 generates a sisterute, which you then pass on to your basal build 07:52.000 --> 07:55.000 so that you're building off of exact same libraries 07:55.000 --> 07:58.000 that you're actually using to deploy on your robot, right? 07:58.000 --> 08:00.000 You can use a bunch of other things as well. 08:00.000 --> 08:03.000 Like, I'm desktop, I'm even, again, rules distorless. 08:03.000 --> 08:06.000 Let's you export out a tarball. 08:06.000 --> 08:11.000 So you can use that as a sisterute or as a component in an image, right? 08:11.000 --> 08:14.000 Which is basically what part of what this is kind of showing. 08:14.000 --> 08:20.000 You can layer up a bunch of different components from the 08:20.000 --> 08:22.000 Docker image right here. 08:22.000 --> 08:28.000 Kind of a distor almost to the run files for your Python environment 08:28.000 --> 08:31.000 as well as for the actual image, right? 08:31.000 --> 08:33.000 So once you pull that in, you have a tarball, 08:33.000 --> 08:35.000 and you can set it up however you want. 08:35.000 --> 08:37.000 You can set up a .img file. 08:37.000 --> 08:39.000 You can put a slap on a bootloader on it, 08:39.000 --> 08:42.000 and then suddenly have something you can just flash on a robot. 08:42.000 --> 08:44.000 And it's going to start up in your conversion, right? 08:44.000 --> 08:46.000 So now you've got an image. 08:47.000 --> 08:49.000 Maybe you've got some deviant packages on top, 08:49.000 --> 08:53.000 containing the actual robot components that you're setting up on this robot. 08:53.000 --> 08:57.000 How do you deploy this on a system that's on the on the road, 08:57.000 --> 09:00.000 or in someone's in a customer's warehouse, 09:00.000 --> 09:03.000 or wherever the how you are deploying these robots, right? 09:03.000 --> 09:07.000 There's a lot of OTA solutions over the updates that you can use. 09:07.000 --> 09:10.000 This is one of them, and it's quite a common one. 09:10.000 --> 09:13.000 As we come in robotics, you can find plenty of others, 09:13.000 --> 09:16.000 but I've personally worked with this one in the past. 09:16.000 --> 09:18.000 But what you want to look for in one of these solutions 09:18.000 --> 09:21.000 is something that has the basic deployment capabilities 09:21.000 --> 09:24.000 and a very common thing you want with this is AB partitioning 09:24.000 --> 09:26.000 so that you can deploy two versions. 09:26.000 --> 09:29.000 And when you do an upgrade, if an upgrade fails, 09:29.000 --> 09:31.000 it switches you back to the previous version. 09:31.000 --> 09:33.000 You want something ideally with modern health checking. 09:33.000 --> 09:36.000 You can do it by the same solution, or separately. 09:36.000 --> 09:39.000 And remote access to this is something that you're able to do 09:39.000 --> 09:41.000 in your specific deployment, right? 09:42.000 --> 09:45.000 So Mender does provide basically all of these. 09:45.000 --> 09:48.000 It makes it in a way that's easy to configureable and changeable. 09:48.000 --> 09:52.000 So it's definitely a solid option for where you're looking at 09:52.000 --> 09:54.000 doing OTA updates. 09:54.000 --> 09:56.000 With that in mind, that is my final slide. 09:56.000 --> 09:58.000 So I want to say thank you. 09:58.000 --> 10:00.000 If you want to, all the slides are going to be uploaded 10:00.000 --> 10:04.000 in the, in the Boston page, but yeah, enjoy. 10:04.000 --> 10:09.000 Thank you.