WEBVTT 00:00.000 --> 00:12.400 I started here, because I have a lot to cover, also unfortunately couldn't get video working 00:12.400 --> 00:21.200 on my laptop, and my robot is up on tail scale, so I'll do the demo after for people who 00:21.200 --> 00:28.440 are interested, I'm really sorry about that, we tried a bunch of times earlier, so let's 00:28.440 --> 00:45.200 get started, so this is my robot, these are some of my other robots, and I also actually 00:45.200 --> 00:52.920 get paid to work on other people's robots, which is just like the best job ever, like 00:52.920 --> 01:08.240 far, I work on big industrial arms, agricultural robots, spot, spacecraft, hotel robots, 01:08.240 --> 01:18.560 it's just, it's great, but let's go back to this robot, this is a hacker bot, it's pretty 01:18.560 --> 01:28.080 spy, it's a small startup in Colorado, called Hacker Bot Industries, the founder is a colleague 01:28.080 --> 01:36.280 of mine in Bernstein, you might remember, no him, from being a founder of the Sphere Robot 01:36.280 --> 01:44.600 all, I don't even remember that, or the Misty Robot, he likes doing social robots, and so he came 01:44.600 --> 01:48.320 about two years ago, we met up at a conference in Boulder, and he's like, I think you 01:48.320 --> 01:54.200 know the social robot, and become like a turtle bot kind of thing, and, you know, the 01:54.200 --> 02:00.800 expressive, and, you know, told me about it, I was like, sure, you know, he's an industrial 02:00.800 --> 02:07.840 crowd funder, and I really, sure, you can't get turtle lots anymore, they're not making 02:07.840 --> 02:17.640 the create three, you know, there's just not a, like a cheap hackable base out there, 02:17.640 --> 02:24.200 everything other robots have gotten to be like two to four thousand, like four or five 02:24.200 --> 02:30.440 thousand dollars, I love the Leo tech guys, and they raised for their latest version, they 02:30.440 --> 02:38.560 raised the price from like $255,000, I was like, okay, great, a cheap robot base, yeah, 02:38.560 --> 02:45.040 sure, signed me up, and then he mentioned that it was going to be like basically a Ruben 02:45.040 --> 02:49.480 off, he kind of kind of just company in China that I was making a Ruben off off that had 02:49.480 --> 02:56.480 a built-in wide R, and it is all integrated into the two-year system, and you just push 02:56.480 --> 03:00.720 button and it wakes up, the first thing it does is rolls around your house and it maps out 03:00.720 --> 03:10.480 the space, you before it goes on clean, and then you can, and that says the map internally, 03:10.480 --> 03:16.480 and I was like, okay, I have to have one of these, you're going to go off and draw this 03:16.480 --> 03:21.080 AI stuff, it's going to have an arm, it's going to be expressive, and I just want it from 03:21.080 --> 03:25.200 the mapping, I mean, if I could, like, just get it from that and figure out how to get 03:25.200 --> 03:34.120 the map out of there, I'd sign me up, and so he helped me out a lot with like, getting 03:34.120 --> 03:40.560 the commands, figuring out the commands and stuff for it, and this is basically what it does, 03:40.560 --> 03:47.320 this is my goal was a one-touch mapping utility robot so that I could map a space and not 03:47.320 --> 03:51.840 have to, like, follow around with a joystick and one hand and tell me up and a laptop and 03:51.840 --> 03:56.640 the other, and I also spend time here. He didn't get tired and you're following it out 03:56.640 --> 04:04.000 of Wi-Fi, and it's just the idea of having this thing, you push button and it goes around 04:04.000 --> 04:10.680 and it does it for you, and then it's usable for other robots, like your roster robots, 04:10.680 --> 04:24.760 I have a lot of really, really wanted to do that. So the thing here is to robot generates 04:24.760 --> 04:33.480 its own onboard map, it auto colors, figures out, it has some sort of algorithm inside for 04:33.480 --> 04:42.000 figuring out where what it think a room is, and then it stores that as a bit map and it's 04:42.000 --> 04:51.800 own custom format, and then you have to have your phone and you get this app called TUIA, 04:51.800 --> 04:59.840 which I guess is like if you're not doing Siri and you're not doing some other home environment 04:59.880 --> 05:06.480 things, you use TUIA, because they control, like they're also trying to, and they have like air 05:06.480 --> 05:17.200 conditioners and rice cookers and window shades, just like everything is hooked into that. 05:17.200 --> 05:20.880 And then you can also, it turns out that there is a plug-in for homocystent, there is like 05:21.040 --> 05:29.920 an integration with TUIA, but I looked at that and it turns out that they're using homocystent 05:29.920 --> 05:37.600 is using the TUIA SDK, which is great except that they want like a lot of money to buy a 05:37.600 --> 05:44.800 license for the TUIA SDK, and then homocystent is also its own lifestyle, which I appreciate, 05:44.800 --> 05:54.720 but I just kind of wanted to get the map off the robot. So that invites some hacking, like 05:54.720 --> 06:06.000 all good open source or open-ish hacking robots, there's a Siri report. Ian has added a 06:06.000 --> 06:15.360 TUIA, not a TUIA, but a TUIA, armed board from Adafruit, on to it, to make the Siri lane facing 06:15.360 --> 06:21.280 a little bit easier, so you don't have to play with the actual wires, but I've done that before, 06:21.840 --> 06:29.280 and basically that you talk to it is you send it hex commands and it spits out data about 06:29.280 --> 06:35.360 like all the bumpers activated, what is the battery status. And this also turns out that there was 06:35.360 --> 06:41.840 a map, a command that they did not document, that was also like displayed map, and this is how 06:42.400 --> 06:49.840 the map gets from the robot, from the flash on the robot over to TUIA. It's all done over wireless, 06:49.840 --> 07:01.440 so we figured that out, and then let's go back one, that led to realizing you can get this map. 07:02.320 --> 07:07.760 This is my apartment, San Francisco, it's a Victorian, so if you know anything, it's basically 07:07.760 --> 07:16.160 it's a single long hallway with rooms off the side of it, and so it looks a little weird, 07:16.160 --> 07:23.120 but that's really like the top part there is entrance to it on the hallway, and then then there's 07:23.120 --> 07:36.400 like a main room, and then there's kitchen, and so that's the map. So we got the map as a bite stream, 07:39.200 --> 07:47.120 off the serial port, and start looking for headers, start looking for repeating patterns. 07:48.080 --> 07:54.640 I actually did some more on the XV11, the old Nito, a wide R for controlling the wide R, and 07:55.520 --> 08:06.160 had it dump, just from, again, serial and interpreting hex codes, got it to dump out a wide R data. 08:07.280 --> 08:14.240 So I was an entirely unused to this, it's kind of a pain, but so we found the header, 08:14.240 --> 08:22.880 and then realized that it is an occupancy grid, because all the thing is doing it with the map is that 08:22.880 --> 08:28.000 it's cleaning, and so it just needs to know where the obstacles, where the boundaries and things are. 08:28.720 --> 08:34.240 So it's actually pretty simple, but converted to gray scale, and we still have, so here's 08:34.240 --> 08:37.040 like the interesting part, or getting into the interesting part. 08:37.840 --> 08:45.040 We're able to convert some gray scale, but it's still encoding some colors. The white part, 08:45.040 --> 08:50.960 the white region is actually part of the image, native coming off of the, coming off the robot, 08:50.960 --> 08:59.520 which is going to be our problem. So it encodes the color for user convenience, but actually when 08:59.520 --> 09:04.240 it goes around and navigates, it doesn't have any concept of events, it's like you're just supposed 09:04.320 --> 09:09.200 to on the phone point, but it doesn't actually really matter operationally. 09:11.040 --> 09:16.160 And this is a little bit different from Ross, where it's just a single layer, and everything in 09:16.160 --> 09:25.040 the single layer, it represents everything. Ross abstracts things like rooms and boundaries, 09:25.040 --> 09:32.080 and user preferences to other layers. You have a global cost map, here at the local cost map, 09:32.240 --> 09:36.960 then you have an inflation layer, then you have, you know, you can set waypoints, and you can 09:36.960 --> 09:48.560 customize it. So first thing I realized is that because Ross encodes white as open to 09:48.560 --> 09:56.400 explore and open, the boundary, just where the map is, is the robot would consider that open. 09:56.640 --> 10:02.960 Ross or all, consider that open. So the entire image is just this open space, and 10:06.000 --> 10:15.040 this is where the conversion starts. I have this media idea, hey, open CV. Open CV detects 10:15.040 --> 10:22.960 edges, it has some, you know, has like camion fine contours, and a, you know, fine poly and a bunch 10:22.960 --> 10:31.120 of other things. And well, as I say, you know, now you have two problems or end problems. 10:34.320 --> 10:39.440 This is not going to be an easy solution, because I realized really pretty quickly, because 10:39.440 --> 10:47.600 candy edge detection is really, it's great, it tries to find an edge for everything. It tries 10:47.600 --> 10:52.560 to find a line or an edge for everything, even the tiniest little feature it thinks is an edge. 10:53.440 --> 11:02.000 And what do you get with light R? You get a bunch of, if you've ever worked with Ross and with 11:02.000 --> 11:09.440 light R, you get a bunch of little spikes, a bunch of little edges just natively. And it turns out 11:09.440 --> 11:14.880 that's actually not a Ross thing, that's a light R thing, because the rays point through and they 11:14.880 --> 11:20.640 all get, oh, hey, maybe that is some area that is unexplored. Maybe that is some area, you know, 11:20.640 --> 11:25.600 you don't want to just throw away things, you take in everything, and hold on to it for later, 11:25.600 --> 11:34.560 and maybe you'll explore that area, or maybe it'll just be this. So that's really just hard to 11:34.560 --> 11:41.120 filter out with candy, you know, if doing a lot of carpet fitting, a lot of tuning, and it still 11:41.120 --> 11:48.400 just didn't work properly, and you're creating boundaries, where there weren't any before, 11:48.400 --> 11:58.080 and I tried a couple other things, like, you know, finding a kernel, to do 12:02.080 --> 12:06.480 finding several kernel operations, and using some of those functions in OpenCV. That didn't 12:06.480 --> 12:11.840 really work either, because it wants to apply to the whole image or to the whole subsection. Now 12:11.840 --> 12:22.160 I'm going to define subsection, so it's just epicycles upon epicycles. So turns out, 12:23.040 --> 12:33.440 if OpenCV says no, go ask NumPy. And if you're familiar with NumPy, and OpenCV are just like this, 12:33.520 --> 12:45.600 they're just like the bestest buddies. And it turns out that to NumPy, an image is just an array shape 12:47.280 --> 12:56.000 of, you know, a 2 by 3 by N matrix, and so anything that you could do with like machine learning, 12:56.800 --> 13:01.440 you know, knowing that work model, training data, where you still have to be out your way 13:01.600 --> 13:06.480 and you're selecting out your parameters and things like that. You can do the same thing with an 13:06.480 --> 13:12.160 image. It just doesn't care. It just shows up as it takes a bit map. It's just an array of data. 13:13.680 --> 13:26.400 So I went poking around in the API, and I found this, NumPy.org, which selects out anything that is a 13:26.400 --> 13:37.280 non-zero value. And that turned out to be one of the answers because you select out, if we go back. 13:40.160 --> 13:49.360 Here, to the gray, now you can just, I can select out where all the boundaries are, and 13:49.680 --> 14:10.160 get that as a separate array and start marching through that. So apply a 6-way kernel 14:11.440 --> 14:15.520 where you're looking at, like, for an edge, you're looking at the left to the right, 14:15.520 --> 14:22.880 up, down, and down the diagonals, and just come into march around the entire inside of the image 14:22.880 --> 14:35.120 and find the boundaries. Get to this. And you get the same map, but now it has black boundary 14:35.200 --> 14:45.920 around it. And now that we've done that, you could out the white, you can invert the colors 14:47.120 --> 14:55.440 because, and just kind of do with it, whatever you need. And so at this point, we've 14:56.080 --> 15:05.200 downloaded the byte stream, converted it to binary, which gives you a bit map. We've flushed 15:06.000 --> 15:11.920 all of the user-defined color regions into a single region, and I've got a border. And so now this 15:11.920 --> 15:22.560 is ready to make available as a cross-cost map. And I found that also, like, this worked well enough, 15:22.560 --> 15:27.120 that actually cleaned up the apartment a bit. And I got a lot more free space, and so now you can 15:27.120 --> 15:35.680 actually through the rooms better, and you can see some artifacts. And so, you know, that's another thing 15:35.680 --> 15:45.520 is, you know, inspired to clean up my room because they actually, this is it. So that is, 15:45.680 --> 15:57.600 buying the demo, I can take questions because this is pretty much about it. This is 15:58.800 --> 16:10.080 up on GitHub for people to look at and play around with. I am actually actually uses to map out 16:10.080 --> 16:17.200 my co-working space in Oakland, California. I work at, I have a kind of appointment there, 16:17.200 --> 16:25.840 as a robot assistant residence at a hardware incubator call, a circuit launch. So it's all up on 16:27.360 --> 16:33.440 up on my GitHub, and then there's the hacker bot industries linked to her if you, they're, 16:33.440 --> 16:44.240 he's still selling hacker bots. And I'm currently also looking for work, always, you know, 16:44.240 --> 16:51.920 contracting life, always looking for interesting projects, interesting companies to help with 16:52.880 --> 17:00.560 their robots with. And come find me, I have cards if you're interested in that. And then also 17:01.120 --> 17:12.160 wanted to pass on some thanks to the Phospham for having me here. I circuit launch family in Oakland 17:12.160 --> 17:21.920 for giving me the space to work on robots and access to tools and 3D printers and the kind of things 17:21.920 --> 17:29.200 that let me build my robots. And then the home robotics club, the Silicon Valley, who I've been a 17:29.200 --> 17:34.720 member of for just like years and years, they got me started on convince me that I could build robots. 17:35.280 --> 17:43.840 I was only a software guy at one point, and then also the character CVC, CV2, R, O, and S, 17:45.520 --> 17:54.400 and also people like you for, I mean, listening to me. And I think with that, I have a few minutes for questions. 17:59.600 --> 18:06.240 Thank you. Definitely come see me afterward if you're interested in looking through the 18:06.240 --> 18:12.480 code or if you want to see it, it actually work as a demo. Hi, yes. It's listening, does it work? Yes. 18:18.480 --> 18:28.240 You know, I've got it. I've got it on there. Yeah, but sure, it's, you know, it's a typical 18:29.200 --> 18:33.440 San Francisco. I don't know if you're familiar with the Victorian style, where it's a really, 18:33.440 --> 18:40.720 really super long hallway. Yeah, it's, it's that. And it's a little bit of a mess, but, you know, 18:40.720 --> 18:51.200 that's where I don't remember a lot. Anyone else? So you also used the Velodine. So did you also 18:51.200 --> 18:56.240 prepare the accuracy between them? I'm sorry, what? You also showed that you used the Velodine 18:56.560 --> 19:04.480 in your apartment? Did you like a variety of accuracy? Yeah, it's, let me go back to that slide here. 19:06.560 --> 19:13.040 So you can see that the Velodine, I mean, it's just like, it's an amazing piece of tech. It's like, 19:14.080 --> 19:19.360 it takes about like maybe 15 or 20 minutes for the hackerbot with the consumer-wide R, 19:19.360 --> 19:27.520 which is like a, you know, like a slick and harpsied R, kind of a lane. It's for it to roll around 19:27.520 --> 19:33.520 and to map that. The Velodine just did this in like five minutes, because the distance, I mean, 19:33.520 --> 19:38.560 this is why it shows up on autonomous vehicles and AMRs and things like that. It just shoots down, 19:38.560 --> 19:44.800 like, I had it like starting here, and it just shot down at the end of the hallway. And so, 19:45.440 --> 19:52.800 didn't, like, five minutes. And you can see the, I mean, I can compare like accuracy in terms of, 19:52.800 --> 20:00.880 like, you know, some meters or, you know, but I noticed that it just, I got much cleaner, just the 20:00.880 --> 20:09.440 plain map. I got much cleaner lines. There's much, it's just much higher resolution, like, 20:09.440 --> 20:17.280 per degree turn. And it just saw everything. I mean, like those raised down there at the bottom, 20:17.280 --> 20:22.080 that's into the back of the laundry room. It saw that, like, it just comes across the, you know, 20:22.080 --> 20:29.440 the second barrier into the big room. And it just saw everything. So it was almost like, 20:31.680 --> 20:37.120 no point in, you really do much more with it, because it's like, okay, I have not been five minutes. 20:37.360 --> 20:42.400 I actually ended up moving, that was for a client project and actually moving it over to my co-working 20:42.400 --> 20:48.080 space, because it turned out my apartment was just like too small. And it wasn't useful for 20:48.080 --> 20:51.680 the kind of data that we were doing or trying to, kind of, data we were trying to get. 20:57.680 --> 21:00.320 Hi, Hugh. Oh, that's fun. Thank you. 21:01.280 --> 21:08.400 My question is, so you, you said that the hacker votes has a shock wrench later, and it's 21:08.400 --> 21:15.600 moved to be able to, to get everything in range, let's say. Right. Do you know how the robots 21:15.680 --> 21:24.960 look at themselves? Um, I think it has, like, it has, like, an onboard IMU, it has the wheeling 21:24.960 --> 21:33.840 co-orders. Um, I, it wants to, like, one of the things that it does after it's done the initial 21:33.840 --> 21:41.600 mapping is that it can, it seems to consult the map that it just made, because they ask you to, like, 21:42.240 --> 21:46.240 consumer device, so they say, hey, put it right here and push the button and say, 21:46.240 --> 21:53.680 want you to start on the base. So it has a little, it has a little doc. Um, so it uses the map that 21:53.680 --> 22:02.240 it just made, and it re-homes itself towards the base, and then it does kind of like the classic 22:02.240 --> 22:09.040 room of docking where there's some sort of an IR, you know, and then it docs itself. And then 22:09.920 --> 22:14.240 it either charges, or if you push the button again, then it goes out and actually, it realizes 22:14.240 --> 22:20.880 it has a map, and it goes out and cleans. Um, so I, so just like my experience happened, like, 22:22.320 --> 22:28.640 tapped into the, um, the serial port while I was doing all of this. Um, but my, my guess is that, 22:30.560 --> 22:36.320 um, it is localized, it knows kind of where it started from, and then it just localizes back to 22:36.320 --> 22:45.440 that, and then it finds the base, stores that somewhere in memory. Um, and then it, it runs with that, 22:45.440 --> 22:53.680 because once it finds the, once the map is done, it finds the base pretty easily, um, just again. 22:53.680 --> 23:00.320 So I think there's like, it makes the map, and then it's kind of like relocalizing sort of, 23:00.640 --> 23:10.400 on the spot. Um, it's all, um, kind of like when, you know, it's a little hard. 23:11.840 --> 23:18.640 Yes, thanks, but yeah, uh, for me, it's, uh, quite impressive that it manages to stitch correctly 23:18.640 --> 23:23.760 all the later information, uh, because for example, on carpet, if you have carpet on the floor, 23:23.760 --> 23:31.520 yeah, it's, it's, um, I, I, I don't want to say other than, um, that's kind of China. 23:31.520 --> 23:38.080 I mean, that's, that's the level of hardware and, and devices that they, they produce, um, 23:39.360 --> 23:44.480 don't ask me about the actual soft, you know, like the user software, but the firmware, um, 23:45.760 --> 23:52.320 that's about average, uh, for devices that I've seen, uh, from them for, like, for that, 23:52.400 --> 23:59.760 for some of the arms, um, the mechanisms and the control, you know, the onboard control firmware, 23:59.760 --> 24:04.400 is actually really quite impressive. Um, and it's like, this is why it's harder to be. It's a consumer product. 24:05.120 --> 24:11.520 It was for 150 dollars, you asked. Um, it's just, why would you be even predicate with that 24:11.520 --> 24:16.960 in the marketplace, um, and just does it? Thanks.