WEBVTT 00:00.000 --> 00:09.600 Okay, yeah, as a tradition, we are doing the infrastructure 00:09.600 --> 00:14.200 of you now, also as a tradition, I was just in Formula 00:14.200 --> 00:18.400 Nitirons of L, so I will leave on the hour at 1635. 00:18.400 --> 00:25.400 So yeah, you clicked, okay, so for those who have been here in the past 00:25.400 --> 00:29.200 in the infrastructure where you will see that this is pretty much the same as 00:29.200 --> 00:34.800 the last few years, and this is by design, because boring is good. 00:34.800 --> 00:39.600 One of the most important things which we did over the last few years and anyone 00:39.600 --> 00:43.600 who wants to clone our conference or anything, you'll see more at the end. 00:43.600 --> 00:48.200 Try and put stability into what you're doing, do not reinvent the wheel every 00:48.200 --> 00:52.600 single year, it might sound exciting and everything, but it is actually good to be 00:52.600 --> 00:58.320 boring, have stuff which works and then just make it work or keep it working. 00:58.320 --> 01:01.920 So as of today, what we have is we have a Cisco ASR which is just 01:01.920 --> 01:05.920 used to be like a super expensive router these days, you can get it for almost 01:05.920 --> 01:10.400 tree, which is most of the stuff, in particular it does the Nit64, we're going to 01:10.400 --> 01:15.040 get that to that in a second, we have a couple of old switches, we have a couple 01:15.040 --> 01:20.000 of slightly less old switches which do 10 gig throughout all of our infrastructure 01:20.000 --> 01:23.040 where we don't rely on your bees. 01:23.040 --> 01:27.400 We have two really old servers which do a few things and we have three more 01:27.400 --> 01:31.240 servers as of recently which run proxmox and then we can run all the different things 01:31.240 --> 01:36.200 on proxmox, all the different services also broken out into different VMs, so it's not 01:36.200 --> 01:44.520 like this huge pile of crap on one single server, we actually were able to segment stuff. 01:44.520 --> 01:48.120 All the monitoring is done, we are permitting this low-key graphana if you don't use any of those, 01:48.120 --> 01:52.200 highly recommend it, I'm obviously biased or we are biased but because we maintain 01:52.200 --> 01:57.880 as in everything but it is really good and we have a T-Gron service where we can 01:57.880 --> 02:04.600 persist data and where we can host longer run in services, basically under the year and 02:04.600 --> 02:08.120 everything, it also like a few of the control service and such. 02:08.120 --> 02:13.240 I forgot if we have an extra IPv6 slide, now we have this at the end, just click. 02:13.240 --> 02:18.200 So for video, do you already want to take video? 02:18.200 --> 02:23.160 So high for me as well, I'm Basti, I'm also part of the infrastructure team here at the 02:23.160 --> 02:24.200 Faustam. 02:24.200 --> 02:29.720 For video, you see that most of the rooms that you've been walked in, there's a tiny little box 02:29.720 --> 02:33.800 here capturing all the videos, trimming it live into the world, hello to my family, they're 02:33.800 --> 02:35.880 watching, I hope at least. 02:35.880 --> 02:41.800 And we do collect everything from those boxes offsite and make it available to the folks that 02:41.800 --> 02:47.400 are not able to join or live stream it and watch it afterwards. 02:47.400 --> 02:55.480 We have roughly more than 1,000 talks that we are producing during each of the events with 02:55.480 --> 03:02.600 those boxes and if you like, you can build them, it's all open design like the PCB sensor 03:02.600 --> 03:09.320 on, it's everything is on GitHub available on our repositories, we have those boxes here, 03:09.320 --> 03:15.960 they send their streams to the renderer farm, the rendering farm, dumps it somewhere on the net, 03:15.960 --> 03:22.520 they'll do the cutting and we do all the pre and post processing until it hits our streaming 03:22.520 --> 03:29.480 endpoints and then maintainers or death room managers or whoever gave a talk can then 03:30.120 --> 03:35.080 open the browser, get an invite and they can just cut their talks and do their thing. 03:35.640 --> 03:43.080 Video boxes look like this nowadays, it's just Linux with some advanced hardware that 03:43.080 --> 03:49.560 there's able to communicate with the rest of the world, nothing very spectacular but it's 03:49.560 --> 03:57.160 built by the awesome video team that we have and yeah I think it's one of a kind that we have 03:57.160 --> 04:01.400 such cool features in there. Next year there will be some new features and I'm not going to 04:01.400 --> 04:08.600 talk about too much what's happening with them but we will use them also to provide you a better 04:08.600 --> 04:17.560 experience in here. Yeah I'm not sure it's quite quite dark that pictures, this is how it 04:17.560 --> 04:24.040 looks on inside, there's a lot of tech stuff, you can eventually see that there's a fuzz 04:24.040 --> 04:29.240 them logo on it because the PCBs are custom made, also those, they're fuzz them logo 04:29.240 --> 04:37.720 within the backside but we try to do everything on our own and not rely too much on commercial 04:37.720 --> 04:43.880 vendors here. If you want to know more details, there's an excellent talk from Martin and 04:43.880 --> 04:49.640 Angle, they gave it last year, highly recommended to watch it, it's produced by one of those boxes 04:49.640 --> 04:57.320 and conserved, the link is behind that QR code, it's not a trap, you can scan it or you can just 04:57.480 --> 05:05.080 check last year's archive and it's not rig rolling, it's not never going to give you up, 05:06.280 --> 05:14.360 no worries. To do such a thing, it's the same with the network and with the video we have very 05:14.360 --> 05:20.840 detailed plans on what to do, when to do who's doing what for for each room, so we make sure to 05:20.840 --> 05:25.880 not waste a lot of time of our volunteers, they're all doing that without getting paid, they get 05:25.880 --> 05:32.600 some food and get a nice shirt, thank you for rolling, tearing by the way and that's it and so we 05:32.600 --> 05:38.120 want to make it as effective as possible and so that's why we try to organize it in a way so we can 05:38.120 --> 05:44.520 run the conference year after year without turning too much through volunteers. This is what 05:44.520 --> 05:49.800 it looks like from inside of the knock or walk network operations center and video operations center 05:49.800 --> 05:55.320 where we see everything that is happening, this was just, but earlier today we have the ability 05:55.320 --> 06:00.280 to have a look at everything that's happening inside the rooms, you see the audio meters here, 06:00.280 --> 06:08.040 we can see if the audio is correct or not loud enough, things like this can automatically be detected, 06:08.040 --> 06:15.240 also the operator see that immediately if something's off, we can then for example just lock 06:15.240 --> 06:21.400 ourselves remote into those boxes and just pull up the lever and get better audio quality, 06:21.400 --> 06:27.560 we can see the near real time, it's streaming the data from the box so we can see everything's 06:27.560 --> 06:36.520 fine, audio is working and if you want to photograph this I will put the slides just also on the 06:36.520 --> 06:42.840 website so you don't need to take photographs of that. That being said, data is captured here, 06:42.840 --> 06:48.920 we send it to a rendering farm, it looked like this, pretty bad in the last years, we improved it, 06:49.000 --> 06:57.640 this is what it looks like this year, now we have better, better conditions, we most of the 06:57.640 --> 07:03.640 things that we've changed with the central switches here, we have now better connectivity and this 07:03.640 --> 07:10.120 is the main thing that changed from last year to this year, you see it, we now have two uplinks 07:10.120 --> 07:17.160 so we're fully redundant with 20 gigabytes instead of 10, there's a lot of people crossing that 07:17.240 --> 07:22.840 room and going there, so having redundancy if someone slips over the cable so it's not a bad 07:22.840 --> 07:27.880 idea to have, so this is the only thing we changed for that year that we got a second fiber that we 07:27.880 --> 07:35.320 run to that switch of the room and are redundant on that side as well. Time lines, Richie, 07:35.880 --> 07:41.400 this is typically your topic, how it used to be and how it's now, so can you go back once, 07:41.400 --> 07:46.600 because this is actually something which we never talked about, you can't really see it here 07:46.680 --> 07:51.960 but this is something which I did a few years ago, I just label onto the hardware what goes 07:51.960 --> 07:55.960 there, because usually it's a lot of panic and people asking what should go where and there's 07:55.960 --> 08:00.360 actually one staff who ever seen the year ask me the same question and I always tell them just 08:00.360 --> 08:09.560 look on the label, like no front or anything, but it is super convenient, like put documentation 08:09.560 --> 08:13.560 into the places where you need the documentation next time, yes it's nice to write it on the 08:13.640 --> 08:22.360 wiki, but like this kind of thing is tremendously important to make stuff boring and just work. 08:23.480 --> 08:28.280 Give yourself this just like cheat code from the past and just document stuff properly in the 08:28.280 --> 08:32.760 right places and make all the labels bright orange because you can see them across the room there 08:32.760 --> 08:40.120 ugly but they work, so timelines, you can go through the details of all of this but the 08:40.200 --> 08:47.400 gist of all what you're seeing here is things get better over time and we get sleep. 08:47.800 --> 08:54.920 If you look here, 2015, Saturday, five in the morning, it was basically go back to the hotel, 08:54.920 --> 09:02.840 take a shower, come back and continue working, try and not do this to yourselves, I speak from 09:02.840 --> 09:09.160 experience, try and prepare and again reuse and make things stable. There's one of the main 09:09.240 --> 09:14.600 messages every single year because we see this again and again and again and last year I visited a lot 09:14.600 --> 09:20.200 of conferences all over the world and you have a few very dedicated people and they're driving 09:20.200 --> 09:28.760 themselves against a wall, stop, think, engineer and try and become better and just finish your 09:28.760 --> 09:34.360 stuff earlier so you can actually sleep and also have some fun. You can just click through the same 09:34.440 --> 09:42.600 principle, you see it is just getting better and we maintain the stability that we actually 09:43.480 --> 09:51.320 do stuff before and not just in time. When here we had stuff literally running on the day and here we 09:51.320 --> 10:02.760 have it almost a month and it once. Next, so for 2026 we had a big plan to reduce stuff and like a 10:02.760 --> 10:09.000 few years ago we made this plan, we are basically done, the road service are just sitting there, 10:09.000 --> 10:15.480 we can just get rid of everything, yeah, need bandwidth and like all of those things and the 10:15.480 --> 10:21.480 point is we actually planned this over several years and we executed over several years without 10:21.480 --> 10:26.440 having too much work because the two of us are based in Munich coming here actually is a lot of 10:26.440 --> 10:30.280 work and yes we have people on site but then it's not always the right thing and something is 10:30.920 --> 10:36.600 just need to plug in and write this one command. It is hard to do stuff off-site to give you 10:36.600 --> 10:43.720 self a good timeline and then just execute over time with the timeline and yeah, we run, next. 10:44.920 --> 10:51.160 And since none of us speaks French, getting remote hands here is extremely critical so 10:51.960 --> 10:58.200 better not do it from Munich and bring something without speaking French, that's another part of the 10:58.360 --> 11:04.120 story. None of the security here is speak English, that is actually a problem for us. 11:07.720 --> 11:15.400 So everything went well until yesterday around noonish we had some Wi-Fi issues, the 11:15.400 --> 11:21.160 only issues we had with the network, at least that we were aware of, we had issues with our 11:21.160 --> 11:29.640 AppStory upstream DNS resolvers and kind of blocked us with due to whatever reason and 11:29.640 --> 11:35.320 this only happened on the 06-4 network that we're providing, we changed the NS announcement 11:35.320 --> 11:42.760 from our internal DNS resolvers to some external, we changed then our internal 064 configuration 11:42.760 --> 11:49.880 and reconfigured our internal AppStory service to Fryfunk Munich instead of the old ones that we're 11:49.880 --> 11:54.920 not blaming here, maybe it was an issue we haven't heard from them, they provide a service, 11:55.560 --> 12:00.760 there's privacy friendly and we just routed the traffic to Fryfunk Munich so pick shout out to them 12:00.760 --> 12:13.320 for providing that and after some testing we were just back online completely on the traditional Wi-Fi 12:13.320 --> 12:18.840 that you're seeing here, there's two SSIDs for those of you who have mentioned have seen it 12:18.840 --> 12:25.320 on their phone, one is called fuzzdom, one is called fuzzdom dual stack, it had other names in the 12:25.320 --> 12:33.160 past, we're not gonna name them, so everything we're trying, we just did some minor re-checks 12:33.240 --> 12:43.800 and then everything went back to normal mode, why are we mentioning that, shout out to the 12:43.800 --> 12:51.160 knock team of ffmook for helping us here, also I'm not sure if anyone of you has seen richies 12:51.160 --> 12:57.080 introduction talk or opening talk for the conference, Fryfunk Munich is also listed of one 12:57.080 --> 13:02.600 of the European alternatives that we have since we're talking about sovereignty, they're listed 13:02.600 --> 13:07.160 there, if you'll free to follow those QR codes, I tried my best with having the fuzzdom local 13:07.160 --> 13:14.680 in them, but they're also available later in the slides and they help us out with their service 13:14.680 --> 13:19.160 which they provide for free and it helped us save our ass and run in the conference. 13:21.320 --> 13:29.400 The upnings, almost like last year, we have sponsors, thank you called for sponsoring the bandwidth, 13:29.400 --> 13:34.760 we see nowadays most people are not using that much bandwidth, I see the same here, you're 13:34.760 --> 13:39.480 paying attention, that's nice, you're not compiling or kernel or something like that or downloading 13:39.480 --> 13:45.240 lots, you're actually paying attention, which is nice, and they have mobile phones of course 13:45.240 --> 13:49.880 and then we have a second uplink from destiny, also shout out to our sponsors and they're 13:49.880 --> 13:56.040 providing us the bandwidth that we can use completely independent from what the university has 13:56.040 --> 14:06.280 is the uplink. One fun fact about our network outage, so we had two times when we couldn't 14:06.280 --> 14:13.880 sell shirts at the info desk, those devices are very picky, but actually there were people who were 14:13.880 --> 14:19.960 running network tests while we had our outage and they commanded us on the performance of our 14:19.960 --> 14:30.040 network while we were dealing with the outage. Part of the moral here is switch to a workaround 14:30.040 --> 14:37.640 quickly and aggressively and then fix stuff and properly engineer and verify everything in the background 14:38.440 --> 14:42.680 and then do the cut over back to a normal production, which in our case we had to do of course 14:42.680 --> 14:47.480 while for iPhone is very privacy friendly, we still are not comfortable with like leaking too much 14:47.480 --> 14:53.720 information off-site, we keep pretty much zero logs or anything and we wipe and everything so we 14:53.720 --> 14:59.880 are very conscious about not generating any data which someone might be interested in, so it was 14:59.880 --> 15:05.800 important for us to switch back, but basically click workaround and then back to proper production 15:05.800 --> 15:09.800 as soon as you can. Next slide please, and also you can clone the conference, but those who 15:10.040 --> 15:17.000 care already took screenshots or photos, so the ASR which we have is end of everything, 15:17.000 --> 15:22.760 Cisco has extended that lifetime a few times already, I get to press this because we are staff, 15:25.000 --> 15:33.000 but it's probably going to be completely deprecated but also we kind of like open source around 15:33.000 --> 15:39.080 here in case you didn't notice, some of you probably did, so the plan is we replace the ASR with 15:39.240 --> 15:45.240 network hardware, probably a register, same as the new distribution we have in other places, 15:45.240 --> 15:52.520 but we want to switch the BGP session to routing the net 640 and SX4 is already, but net 64 15:52.520 --> 16:00.680 to open source software instead of running it on close source hardware, we will have the network 16:00.680 --> 16:06.200 hardware as backup for the upstream, just in case something goes boom, but most likely and keep us 16:06.280 --> 16:12.280 honest, next infrastructure review, we will be able to just be on open source hardware, 16:12.280 --> 16:17.240 well not hardware, but software for probably the entirety of the routing and everything. Next slide please, 16:18.600 --> 16:24.760 the other thing, over 10 years ago I had a really stupid idea and I said hey why don't we switch the 16:24.760 --> 16:30.600 main network to net 640 and it was like yeah I do wait a minute okay, for over 10 years I thought 16:30.680 --> 16:36.440 this was the second large major network conference which did this and I have been informed 16:36.440 --> 16:41.640 on last Friday there was actually the first worldwide where we did this and we found eight 16:41.640 --> 16:48.120 ton of bugs in various distributions in the Linux current in a lot of software because this is 16:48.120 --> 16:56.040 faster, so we actually like I have friends that drive in such we actually measureably improved the 16:56.040 --> 17:03.320 I need to leave soon, the IPv6 distribution in software and like support and everything and even 17:03.320 --> 17:12.440 like the adoption on the internet. The plan for next year, the primary network will be IPv6 only, 17:12.440 --> 17:17.720 again keep us honest on this one and we probably start switching out the names of the dual stack 17:17.720 --> 17:24.200 one just so we prevent people from auto connecting to really forest people to to look at IPv6 17:24.200 --> 17:29.560 more currently the main ones and pre-empting the job so I'm going to make it right before the 17:29.560 --> 17:36.280 closing is a get up which is still IPv4 only so if anyone from get up it's listening next year 17:36.280 --> 17:44.600 one of your largest users is going to be IPv4 only, in a change we can just put the scene name 17:44.600 --> 17:58.440 and just redirect to codeberg which supports IPv6, the other thing, the other thing chat.fosem.org is also 17:58.440 --> 18:04.760 IPv4 only as of right now so keeping ourselves honest by next year we expect this to be IPv4 18:04.760 --> 18:11.160 and IPv6 as well and also we are probably going to make a third SSID to just be able to encrypt 18:11.160 --> 18:16.040 traffic we will probably make this IPv6 only as well. I think we are done and have some time 18:16.040 --> 18:25.400 for questions but I need to run really soon. Thank you. Any questions? 18:32.760 --> 18:39.480 What's the rationale behind having a cluster of think pads instead of a big multicore server 18:39.480 --> 18:44.040 like a threadwer bar or something like that? I'm sorry I didn't understand. Oh sorry. 18:44.040 --> 18:52.280 Why think pads instead of a big server? Why think pads? So the question is why do we have 18:52.280 --> 18:58.600 think pads for the render farm? Several reasons just like splitting domains so the blast radius 18:58.600 --> 19:03.560 of any outage is smaller but also we wanted a back rebate back up we wanted to have different 19:03.560 --> 19:07.880 machines back then the ms and such were really brittle and hard way which could do this was 19:07.880 --> 19:11.720 really expensive we wanted to have a keyboard and this place kind of nice in the way as well 19:11.720 --> 19:16.920 there's units which they make which have all of this in that laptop and that's basically why we just 19:17.960 --> 19:26.680 took a pile of laptops. All right did we trip the breakers in the knock or all the places again? 19:26.680 --> 19:34.280 No this was part of last year's post-mortem to first go to the knock, secure the 400 volt output 19:34.360 --> 19:38.920 and put everything there instead of the normal circuit that we're using the years before because 19:38.920 --> 19:44.120 we learned sometimes we learn and then we apply that. Excellent question by the way. 19:46.280 --> 19:52.600 And there's a guy who bought the 400 volt distribution boxes I'm machine sitting down there 19:52.600 --> 19:54.600 because this is the idea to do that. 19:55.240 --> 20:06.440 Hi great conference thank you. Me being network engineer is the special reason for the 20:06.440 --> 20:11.640 MTU being 2004 800 on the IPv6 only network. 20:14.680 --> 20:23.400 That's the thing is that we don't own the hardware and we don't operate it we just got the Wi-Fi hardware 20:23.720 --> 20:29.160 and the Wi-Fi hardware and we just get the Wi-Fi sponsored by the university 20:30.680 --> 20:35.960 and we get whatever day push over the wire and we just have an interface where they throw out some 20:35.960 --> 20:38.760 wheelons in our direction and the rest is in our issue. 20:44.920 --> 20:50.920 Which unfortunately need to leave because he's got another talk in minus three minutes but I'm still 20:53.640 --> 20:59.720 available. You mentioned earlier on software to find storage curious what you're using. 21:00.840 --> 21:08.360 We used built-in safe that comes with proxmox we switched to proxmox a few years ago and we're using 21:08.360 --> 21:15.960 the built-in safe nice shot by the way. And of course we use safe we have folks on our team 21:15.960 --> 21:21.000 that also do software-defined storage in their day job but we agree on using stuff because it 21:21.000 --> 21:25.640 just comes with betterly included when you're using proxmox and it's yeah. 21:27.000 --> 21:33.000 Thank you for explanation. How much infrastructure and help does a university actually contribute? 21:34.920 --> 21:40.920 The infrastructure it's one half size drag that we own everything in there is owned by us. 21:42.040 --> 21:48.120 That means there's a fuzzdom vis-a-vis this is like the club fuzzdom where all the hardware 21:48.120 --> 21:53.960 also belongs. We get the app links we are allowed to use some parts of the infrastructure. 21:53.960 --> 22:01.080 If you look at the lectern here's an Ethernet port over the weekend they turn the wheelon to our 22:01.080 --> 22:07.080 wheelons that we agreed on and this is actually what we get from ULB thankfully also the access points. 22:07.080 --> 22:12.360 You see one two three access points all of that we get from ULB. 22:13.000 --> 22:22.440 Years earlier we brought our own stuff and did wild cabling these years are gone. We can share 22:22.440 --> 22:28.280 the infrastructure and with them which is really really nice. Where's the microphone? 22:29.720 --> 22:37.000 Ah, it's the app, okay. Hi have you considered using BPP FDIO SDR router instead of the ASR? 22:37.000 --> 22:40.760 Yes we have. And will you use it? We're gonna see. 22:42.680 --> 22:50.920 The things we had discussions two years ago around VPP with the people of VPP here on site. 22:52.920 --> 22:59.800 But that time we needed to fix other things in our setup. We're actively working on building 22:59.800 --> 23:07.320 a build of materials that we would need to do that all in server without relying on any vendor 23:07.960 --> 23:13.480 or whatever would come to mind and trying to build out something that could work. 23:14.520 --> 23:20.680 But we just started yesterday. So be with us. But the idea is ideally to go in that direction. 23:20.680 --> 23:28.680 Yeah maybe add a other question. Do not six for as you said that will be that was for some time 23:28.680 --> 23:36.600 of site but variables. Because the DNS six for didn't work. The DNS broke the DNS six for 23:36.680 --> 23:42.760 broke. It was only the DNS six for part that broke. And that's why we did the people with 23:42.760 --> 23:50.760 not six for experienced issues with IPv4 only systems. So you removed the not six for from the 23:50.760 --> 23:56.600 ASR to some somewhere. No we fixed the DNS so the the not six for wasn't the problem. It was always 23:56.600 --> 24:02.200 running. It wasn't usable. It was like that you moved the not six for to elsewhere. 24:02.840 --> 24:08.040 Then it was a typo on the slides. Okay. It's okay. There's one guy over there right in your 24:08.040 --> 24:14.600 hands for longer time. Last question. Okay we need to get we get kicked out here. 24:18.840 --> 24:25.480 I'm interested in using the video capture boxes in the rooms for other volunteer events of 24:25.480 --> 24:31.640 less technical nature. Do you know if there is any organization that would take some money and 24:31.640 --> 24:38.680 make me one. You can rent from from us for for the nation for example. This is one of 24:38.680 --> 24:45.240 things. There's also event in fra from the Chaos Computer Club CCC. I feel like they also do similar 24:45.240 --> 24:51.320 things with infrastructure that you need to run an event. But if you're interested in those boxes 24:51.320 --> 24:59.720 for other non-commercial conferences, we typically ship them to them for small donation and 25:00.200 --> 25:06.200 get them back hopefully in one piece. If you're interested just come talk to us. Okay thanks. 25:06.200 --> 25:12.040 Yes absolutely non-commercial. Thank you. That being said that was the last question. Thanks for 25:12.040 --> 25:24.760 attending Faustum. Have a great rest of the day be excellent to each other and maybe join 25:24.760 --> 25:28.840 the closing sessions in Jossar. It's just down the hallway and then on the right.