WEBVTT 00:00.000 --> 00:11.560 Alright, welcome everyone. I'm back. If you were here in the room before as well, I just 00:11.560 --> 00:16.760 had a talk like an hour ago. But now I'm going to give a talk again. This time about my 00:16.760 --> 00:22.640 summer vacation of last year, I fell into a deep rabbit hole of a reverse engineering 00:22.640 --> 00:28.760 theme of FES and ESXI kernel. And I thought it'd be fun to show you my vacation pictures, 00:28.760 --> 00:33.560 so to say. Quick introduction. I have a lot of slides. I didn't learn anything from 00:33.560 --> 00:39.680 views, but it's a day. So I would go a little bit quickly. Very quickly. I work for 00:39.680 --> 00:44.800 Steve. For 20 years now, I'm a security researcher. I do R&D, a version nearing development, 00:44.800 --> 00:52.600 author of DICEX, which to talk an hour ago was about. And what FES is something we support 00:52.640 --> 00:59.200 and DICEX, and something I spent quite a bit. Quick glossary. So DICEX, I'm not going 00:59.200 --> 01:05.680 to go into DICEX. So I'm going to summarize the talk from hour ago in one sentence. It's 01:05.680 --> 01:12.320 a system investigation network and it's impure Python and it's pretty good. You try it. 01:12.320 --> 01:18.400 A choir. It's a tool that's built on DICEX that allows us to collect artifacts from all kinds 01:18.400 --> 01:26.140 of different systems. EZXI, if you're not aware, hypervisor runs on the on bare metal. And 01:26.140 --> 01:31.840 FEMFS is kind of like the proprietary file system that EZXI uses to do it like, like, 01:31.840 --> 01:39.880 multi-host file systems. And you know, I'll file a file locking for that and stuff like that. 01:39.880 --> 01:45.400 FEMFS 3, this will be important later, introduced directories, which is very exciting. FEMFS 01:45.400 --> 01:50.720 5 and 6 are largely incremental improvements on FEMFS 3. And why it is important, you'll 01:50.720 --> 01:58.720 see later. So I think this is like a few years ago, I think like 2018 or something like 01:58.720 --> 02:02.800 that, we were already kind of playing around with hypervisor based data acquisition, but 02:02.800 --> 02:08.520 it was very cumbersome. So basically what we would do, only in specific configurations. So 02:08.520 --> 02:13.600 if there was a hypervisor that was connected to a storage network, we would slide in an 02:13.600 --> 02:18.280 additional server, basically, attached to the storage network, mount the storage that 02:18.280 --> 02:24.600 the data store was on for the hypervisor. Use the open source available FEMFS or FEMFS 02:24.600 --> 02:29.480 6 tools to mount the data store and then run the acquired to against the virtual machines 02:29.480 --> 02:35.460 on that data store. However, there is a slide problem with that, which is that FEMFS 02:35.460 --> 02:41.980 tools is pretty broken. There's a lot of bugs in there, which means we got a lot of data 02:42.020 --> 02:47.380 corruption, missing data, just straight up for the machine, so we couldn't acquire, which 02:47.380 --> 02:53.940 is pretty bad if you're trying to investigate an entire network. And it's on my 02:53.940 --> 03:02.580 tank, which is not good either. So armed with item, I thought like while how hard 03:02.580 --> 03:10.060 could it be. Few like a year later, so we had like a job that was like, hey, we have 03:10.060 --> 03:18.660 this rate set up. It's trashed, please recover my data. And it was of rate of FEMFS 03:18.660 --> 03:24.900 data store and to be able to kind of aid my job a little bit easier on recovering the 03:24.900 --> 03:33.060 rate configuration. I just built some tools to interact kind of like peak at FEMFS structures 03:33.060 --> 03:37.900 and to kind of like calculate some offsets, so I can see if I got the rate configuration 03:37.900 --> 03:43.980 and the striping right and stuff like that. So I built some special tools and kind of my 03:43.980 --> 03:47.900 high level understanding of FEMFS at that point, looked a little bit like this, so you have 03:47.900 --> 03:54.660 a disk and then you have your regular position table and then you have like FEMFS specific 03:54.660 --> 04:01.460 LVM implementation that's on some offset or some position. And then within that you kind 04:01.460 --> 04:06.080 of like have FEMFS or you have your you know your metadata files kind of similar to how 04:06.160 --> 04:12.720 NTFS has like MFT and use engine or stuff like that. FEMFS takes kind of like a page out 04:12.720 --> 04:18.920 of that book instead of being more like other unit file systems which all have like just 04:18.920 --> 04:24.640 data structures on disk. FEMFS takes a approach of data structures are in files or the metadata 04:24.640 --> 04:30.400 structures are files, so to say. And then if you interpret those, you can just refer to machines 04:30.440 --> 04:38.480 and read your very exciting directory structure. So I wrote a implementation is the along the 04:38.480 --> 04:44.240 very short summary of that. I was like, well, I wrote all these inspection tools, might as well 04:44.240 --> 04:49.280 pull through and make my own interpretation. I took a little bit of FEMFS tools a little bit 04:49.280 --> 04:53.600 of Ida because what I didn't have that much time to rob the implementation, so I was like 04:53.680 --> 05:00.640 just do it quickly. I fixed some of the most obvious FEMFS tools bugs like they were just like 05:02.960 --> 05:08.960 unit for the 64 and I was like unaligned of like some offset, so sometimes the like a few 05:08.960 --> 05:14.400 bytes would be out of bounds of the field structure and then it would calculate the entirely wrong 05:14.400 --> 05:20.240 offset for some file. And I'd have some to do this for the future of course because you know 05:20.320 --> 05:28.240 whoever finishes all the products right away in first time. And at a benefit that we actually got 05:28.240 --> 05:34.000 with this is I integrated FEMFS into disk target, which got like the big framework and because 05:34.000 --> 05:41.040 acquire is built on a disk target, this means that acquire also automatically gets support for 05:41.760 --> 05:48.160 FEMFS. And it just happens to be that if you're on an ESXI shell like on SSH or just like the physical 05:48.240 --> 05:53.520 shell, you can actually read the block devices that the data store is stored on. The data is 05:53.520 --> 05:57.760 local storage. The design is a whole different beast. We're not going to go into that, but it was like 05:57.760 --> 06:04.160 local or I suppose the storage, you can actually just access the block device, read it, you can 06:04.160 --> 06:11.360 even write to it if you want exciting things there. But this means that if we just run an executable 06:11.360 --> 06:16.160 on the system, we can just open a block device as part of the FEMFS tools structure as 06:16.160 --> 06:20.240 and just start reading files straight from the block device, kind of like how things like roll 06:20.240 --> 06:27.520 copy work on a Windows as well. And I was doing the world until you open source it and someone 06:27.520 --> 06:34.400 comes along and says your shit broken and then you're pretty sad because that means you have to 06:34.400 --> 06:42.640 start looking into it again. Okay and I got a pretty nice you know I got a stack trace so that's 06:42.720 --> 06:49.920 pretty good. But someone came along and was like you know if I just create a big data store 06:49.920 --> 06:54.960 and I create two directions in there and I try to list the second directory it breaks. So that's 06:54.960 --> 07:00.800 pretty big problem because you know 65 terabytes is not that big in the sense of of big 07:01.600 --> 07:08.320 music side deployments. So you know we do have to take it seriously because we ourselves 07:08.400 --> 07:14.720 will probably encounter this pretty soon as well. So I started investigating. We talked on this 07:14.720 --> 07:23.120 word. I spent some time like setting up trying to get a reproducible case. Luckily this guy 07:23.120 --> 07:28.880 just like really shares some of his his workflow and like what he thinks is wrong as well. 07:28.880 --> 07:35.120 Rich aids in my investigation as well. I have some suspicion of what might be wrong at this point 07:35.200 --> 07:42.080 but not quite sure. But I am able to reproduce the case like just a very bare bones 07:42.080 --> 07:47.280 reproduction and it turns out if there's any file system that's just bigger than 16 terabytes 07:47.280 --> 07:55.120 then it just starts breaking for some reason. So I started noticing this specific symptom. 07:56.400 --> 08:02.240 Well just like that for everything over 16 terabytes it just breaks and reading like the small 08:02.240 --> 08:08.240 block system resource file. It just starts giving me the wrong data and in FEMFS like 08:08.240 --> 08:12.240 file blocks are allocated either large block or small block most of the things are small blocks 08:13.600 --> 08:20.400 like directory information is small blocks but then you know like FEMDK data would be a large block 08:21.920 --> 08:26.000 and you know I'm trying to read a directory content which would be in the small block file 08:26.560 --> 08:30.480 and if that doesn't work or is it gives me the wrong data I'm starting to parse the wrong directory 08:30.800 --> 08:37.600 information and the cause is who knows what I don't know yet. So but at least I can reproduce 08:37.600 --> 08:43.600 you know I called hide between or behind you know works on my machine. So I just start inspecting 08:43.600 --> 08:49.680 my own code read start combing through which and you know trying to find anything that might be wrong 08:49.680 --> 08:57.360 here and you know adding debug frames you know debug frames trying to see if I can find out 08:57.520 --> 09:02.560 what's what's wrong. I did find a description see but I can't explain it. So I start 09:02.560 --> 09:10.160 for version engineering FEMFS again still can't explain it so I'm like starting to go a little 09:10.160 --> 09:14.880 with deeper version engineering again and like at this point I'm basically re-implementing my 09:14.880 --> 09:20.960 own implementation from scratch again but I still can't explain actually what's wrong. Like at 09:20.960 --> 09:26.240 this point I've just re-version engineering and read create a whole new implementation scratch 09:26.320 --> 09:32.000 up until the point where I can start reading directories again and it's still still wrong and I'm 09:32.000 --> 09:37.360 I'm pretty sure I got it right you know this time looking at comparing it against item but still 09:37.360 --> 09:47.200 I'm just I'm just way off. So what does bug actually look like? Well you know just this hypothetical 09:47.200 --> 09:54.400 hack stump of the small block file. This is what I would expect from reading from these offsets 09:54.960 --> 10:05.360 and this is what I get. So spot the differences. I'm just like 0X1, 6 years off for some reason 10:05.360 --> 10:14.080 and I have no idea why. I do report back to the initial bug report I like hey I have to 10:14.080 --> 10:20.320 determine this. I start to be re-implementing it. I still really configure out why I'm 10:20.320 --> 10:25.520 depth many by itself. It's pretty big. So it's not like off by one. It's off by like a megabyte 10:25.520 --> 10:32.320 really a bit more even. And I even found out that like there's some debug logging you can enable 10:32.320 --> 10:39.600 on EZXI host as well and you can log like what the physical offset would be of a file read 10:40.720 --> 10:45.120 and it's the same. So I'm like I'm pretty sure my calculation is correct. I'm pretty sure 10:45.920 --> 10:51.200 I got all the math right. I got all the offset's right stuff like that. But for some reason 10:51.200 --> 10:55.200 if I just compare my data, if I just hex dump it I'm still just so many by itself. 10:59.600 --> 11:04.960 So I'm like I want to re-reverting near this further because I just like okay well I'm just 11:04.960 --> 11:10.000 going to you know not stop until I 100% lay where everything in Ida. I didn't do that. I'm 11:10.000 --> 11:18.320 not that crazy quite. But I do want to make my re-reverting near my life a lot easier. And having 11:18.320 --> 11:23.760 anything with symbols or debug info is called when you're doing re-reverting nearing. So I often 11:23.760 --> 11:30.240 just try to find you know scrap something together for public sources. And one of my favorite 11:30.240 --> 11:36.160 things to do is use five-stotal retrohands and I just copy pay some strings, run in strings, 11:36.160 --> 11:42.720 I find in binaries and I just run a retrohand and see if I can find anything that has any 11:42.720 --> 11:49.280 reference to these symbols or these these functions. I got a few matches. Very exciting. 11:50.480 --> 11:57.280 Unfortunately the first one is myself. The second one is myself as well. But the third one 11:58.000 --> 12:02.480 the third one is interesting because that says it's an elf. So it's a shared library. 12:02.880 --> 12:10.480 So what is this file? This magical file which has a reference to this string. It's an elf 12:10.480 --> 12:16.720 32 binary compiled with gc2 which is old and while I was making these slides I was pretty curious 12:16.720 --> 12:25.440 how old is that. It's old I mean. And I noticed so on the bottom right you see like some strings 12:25.440 --> 12:33.120 that were in this binary and they were in sections named dot-step and dot-step. And I was like 12:33.120 --> 12:38.400 what is this? And I was already half way implementing my own parser for this. First try 12:38.400 --> 12:44.240 throwing into idah or vidra unfortunately they didn't recognize it. And I was like already 12:44.240 --> 12:50.800 making some Python code to parse this into some some some readable format turns out 12:51.280 --> 12:58.240 this is like a stabs debug format which to my knowledge is pretty old like three days 12:58.240 --> 13:05.360 12 but you can still dump it with object dump minus g and you get like a nice sea header file 13:05.360 --> 13:10.480 as output that you can just you know inspect or throw through do whatever you want. 13:11.920 --> 13:18.800 The binary itself turns out to be some random binary from e63 installation so it wasn't called 13:18.880 --> 13:25.200 e6i back then it was called e6 and it was like an old rpm based version of e6 13:26.320 --> 13:30.880 before they had their own fancy kernel and they were actually like Linux based and they just 13:30.880 --> 13:36.480 shipped a whole bunch of rpm that they installed on the Linux based and it turns out that some 13:36.480 --> 13:44.080 of those old versions actually shipped with debug builds. So the most same thing to do after that of course 13:44.160 --> 13:50.000 is to start hoarding e6 installation media and start figuring out what is the range of versions 13:50.000 --> 13:56.880 that actually contain debug symbols. So this is my target and I with my three greatest friends 13:56.880 --> 14:02.720 five total instant archive and Google good working I tried to collect as many 14:03.840 --> 14:08.320 installation media as possible. I get quite a lot quite a lot of all versions as well. 14:09.280 --> 14:15.120 This screenshot is like it doesn't quite do just as how much time this actually cost 14:15.120 --> 14:20.800 probably took me like a week maybe like we can have to actually collect all of this like find all 14:20.800 --> 14:28.000 of these things. So many of your rails just broken don't work anymore but I got quite a lot. 14:30.000 --> 14:34.400 I also while doing all this searching for all these symbols I stumbled upon 14:35.280 --> 14:42.720 three interesting well two blog posts and one interesting presentation which specifically talk 14:42.720 --> 14:50.720 about something called VMware Workbench and I mean the title says gets symbols like I want this. 14:50.720 --> 14:55.520 I want to get symbols as well but you know then it also says authenticate which VMware use 14:55.520 --> 15:00.400 credentials you know download the VMware Workbench okay okay okay and then you can just have an 15:00.400 --> 15:07.600 RPM with with with symbols and like this is great I want this as well but then you know there's 15:07.600 --> 15:11.680 a screenshot as well what is what look like and you can see like VM kernel like all the builds 15:11.680 --> 15:16.240 like 620 debug symbols and fade away it is great but unfortunately reality caught up to me 15:18.000 --> 15:24.080 brought home you know acquired VMware and well everything's down you can't get any of this stuff 15:24.080 --> 15:32.800 anymore ever it's unfortunately lost and to devoid so this got me pretty sad but I tried emailing 15:32.800 --> 15:41.200 everyone involved in these in these blogs as well and fortunately the author of the ZDI blog 15:41.200 --> 15:49.760 actually had a copy of one of these files at least or like 203 and he was able to pride them to me so 15:49.840 --> 16:02.320 I'm very very very gracious for that so the TLTLDR turns out that also ESX3 all binaries included 16:02.320 --> 16:09.840 debug symbols so like to stab debug format and it turns out that VMFS3 was introduced in ESX3 16:11.360 --> 16:16.640 and because VMFS3 is the version that well like supports directories like it's the pretty 16:16.640 --> 16:23.680 big rewrite of VMFS but fortunately like VMFS5 and 6 are largely incremental improvements on 16:23.680 --> 16:29.840 a VMFS3 so I can still use these symbols to get a pretty good base understanding of how 16:29.840 --> 16:37.120 VMFS works onto ESX3.5 there are debug versions of all binaries so not quite symbols but still we 16:37.120 --> 16:42.160 get debug assertions a lot of debug prints not really helps out in a version game as well 16:42.320 --> 16:53.040 the data that was lost on the VM workbench kind of like fortunately I found out in the end 16:53.040 --> 17:00.160 it only contains like the kernel so like the base kernel binary and you don't get any symbols 17:00.160 --> 17:05.840 or debug versions of any of the kernel modules and VMFS implemented in a kernel module so I 17:05.840 --> 17:13.040 wouldn't even really had that much at a benefit from having these but from the one I did get from 17:13.040 --> 17:21.840 from Reno I was able to get like a few slightly more modern like constants or structured 17:21.840 --> 17:27.280 definitions but only like one or two instead of you know the gold mine that turned out to be the 17:27.280 --> 17:34.240 audit debug symbols so you actually get quite decent symbols from this you get like four 17:34.240 --> 17:42.320 unions as well and this just made refer to engineering so so much easier because instead of having 17:42.320 --> 17:48.160 to guess you know like what the layout is or or an idea you know how how some instructors are 17:49.120 --> 17:56.320 are referenced I just know that you know a disk block is based the union of all these different things 17:57.360 --> 18:02.880 and a file descriptor you know has like a lock block a mena block and then a union of whatever comes 18:02.880 --> 18:08.080 after that and that just makes all the very engineering so much easier and making your own implementation 18:08.080 --> 18:17.520 so much easier as well so armed with this new knowledge turns out so I kind of like tried to 18:17.520 --> 18:26.640 look how much symbols I can actually use so my approach here was to just take a more recent version 18:26.640 --> 18:30.960 of the binaries so I got a recent version of LVM, recent version of the MFS and then I just try 18:30.960 --> 18:36.400 to kind of like match the symbols from the old versions and like have a kind of like side by side 18:37.200 --> 18:42.240 version of the engineering so I do reverse engineer like the most modern version of the MFS but 18:42.240 --> 18:48.080 with the aid of like having a debug symbols of an older version but I still still do a lot of 18:48.080 --> 18:54.320 reversioning from stretch of course but luckily there is this file if you open it there is a Python 18:54.320 --> 19:01.200 file in which there are like there's like a Python file which has some structures copy-pasted in 19:01.200 --> 19:07.040 there from like the the file system header it's not allowed but it does help a little bit with 19:07.040 --> 19:15.360 finding out some newer fields on newer version of the MFS. Long story to you know I thought 19:15.360 --> 19:22.480 this talk was about fixing a bug so I in my completely implementation I was only focused on 19:23.120 --> 19:30.240 the MFS and when I started you know when I finally had all the symbols I started with a version 19:30.240 --> 19:35.440 of engineering everything using the symbols I was like hmm I haven't really looked at LVM yet 19:35.680 --> 19:43.280 maybe the problem is there and so I started reversionering LVM and suddenly I noticed something 19:43.280 --> 19:48.080 in that implementation and I took so many shortcuts in the LVM implementation because that's kind 19:48.080 --> 19:52.320 of the part where in the beginning I said I took some from theme festivals and I basically took the 19:52.320 --> 19:57.280 LVM implementation of theme festivals and that's I I shouldn't have done that I should have 19:57.280 --> 20:03.680 looked a little bit closer into how LVM works because I would have saved me a lot of time here 20:04.000 --> 20:12.400 so I kind of like rewrite LVM as well and as a bonus I actually support like great 20:13.280 --> 20:22.240 configurations of of theme festivals now too so what's the bug this was like my previous understanding 20:22.240 --> 20:30.080 of theme festivals turns out in LVM you actually have a bunch of extents with wood and 20:30.240 --> 20:36.320 you know a gap in between and then theme festivals the file system layer is kind of like overlaid on 20:36.320 --> 20:44.400 top of that so it is immediately explained like the gap or the offset I was wrong with I'm reading 20:44.400 --> 20:50.240 off as files so what I'm still doing in either this point I fixed the bug you know I could have just 20:50.240 --> 20:54.560 made a quick fix in my old implementation but I was like I would just see this through I'll just 20:54.560 --> 20:59.520 finish the entire thing and I rewrite everything from scratch take me about a month a little bit 20:59.520 --> 21:05.360 more maybe but at least this time I know I have everything verified across my own 21:05.360 --> 21:12.320 version engineering instead of you know like basing it off of other people's research not to 21:14.400 --> 21:18.640 disrespect that of course like it's so it helps so much in getting your initial understanding of theme 21:18.640 --> 21:24.160 festivals but you do see like a lot of shortcuts taken in the other implementations 21:24.800 --> 21:34.240 so the fruits of my labor is that in the end I wrote a bunch of code and it got merged 21:34.240 --> 21:40.800 that's good some poor soul had to review this unfortunately and so what's next I'm finally 21:40.800 --> 21:46.560 done not quite so I'm kind of like working on a blog that's report on theme festivals because 21:46.560 --> 21:50.880 I think it's a waste to keep like the knowledge to internal knowledge of actually how theme 21:50.880 --> 21:56.480 festivals really works on at a hoot to keep that the log in a pie file and you know some 21:56.480 --> 22:04.080 either databases it's not doing it much justice so there will be like a bigger report on like 22:04.080 --> 22:08.640 exactly what are like the forensic implications of the MFS exactly what are the data structures 22:08.640 --> 22:14.160 how are they how do they relate to each other stuff like that in the future and that's it there's 22:14.320 --> 22:16.320 no next slide 22:32.560 --> 22:38.400 the question was I use item could I have done this with any like other open source 22:38.400 --> 22:47.600 the compilation tool yes I just personally prefer item is what I used a lot before for example 22:47.600 --> 22:53.520 Keter I was open source and before like primary ninja I was quite as good as it is these days 22:53.520 --> 22:58.240 and I've kind of like just I have my muscle memory in item so whenever I need to get something done 22:58.240 --> 23:02.400 quickly I just were like big I'm just more comfortable in that item 23:08.400 --> 23:19.280 no there's stickers here if you want to buy a way okay thank you