WEBVTT 00:00.000 --> 00:28.600 We're looking forward to bringing you some entertaining little things. 00:28.600 --> 00:35.720 We are part of the Heinlein group and we'll show you some of our things that we are. 00:35.720 --> 00:42.440 The talk is going to be about lessons learned and some interesting tools. 00:42.440 --> 00:46.360 Yeah, I think we'll do the intro just after us. 00:46.360 --> 00:51.800 Yeah, part of Heinlein group, we had a talk from Pascal earlier who was introduced already, 00:51.800 --> 00:59.760 so we have a couple of different compartments here and we are part of the Heinlein 00:59.760 --> 01:07.560 itself and they're the IT baratung baratung means consulting, so yeah, that's us and we also 01:07.560 --> 01:17.640 have a couple of other things, services from various from academy to open talk, which I think 01:17.720 --> 01:25.000 is holding talks and wells and yeah, definitely open cloud, we're really eagerly waiting 01:25.000 --> 01:38.600 as well for it to release as this is a nicely asked and yeah, as well as mailbox, email provider, 01:38.760 --> 01:49.320 also part of the thing and so I would say, yeah, do pass by, have a look, especially now and here 01:49.320 --> 01:55.320 we are at our talk. So we've been doing mail at Heinlein, I've been there since 2019, 01:55.320 --> 02:05.160 Linux, I'm doing since the end of the 90s and we are putting large scale or various kind of email 02:05.240 --> 02:11.560 infrastructures out there and yeah, would like to share something, so maybe you want to introduce 02:11.560 --> 02:19.320 yourself? Yeah, hi, I'm casting, I'm doing nearly the same two years longer at Heinlein 02:19.320 --> 02:26.120 Heinlein, I would say. So the infrastructures were small sometimes, sometimes big, so they are 02:26.120 --> 02:32.840 growing more and more, so we have some pictures from the old times at least one, yeah, let's see. 02:35.640 --> 02:45.080 Yeah, so Jaymap, yay, we are looking forward, but in our day today, there's it's not there yet, 02:45.080 --> 02:54.360 we're looking forward for the 100 this year, soon, soon and then yeah, a lot of things are 02:55.640 --> 03:02.280 bound to that, so maybe next year, I hope this year. Yeah, that would be Jaymap and 03:05.640 --> 03:11.000 that's where it all started, we had our critical infrastructure below our desk and that's what the 03:11.000 --> 03:20.440 mail admins were taking care of and it was all, you know, in your own control, but times have 03:20.440 --> 03:30.120 changed. So this was one of my first experiences as a coinciding, so an appointment out of the house 03:30.840 --> 03:36.840 called from from a customer, okay, yes, it's email, server under his desk, it's not 03:36.840 --> 03:44.840 booting anymore, so I've grabbed a car and was driving to to this customer to fix this email server. 03:46.120 --> 03:54.760 More or less, since then, we we have done a little bit more for a high availability and, yeah. 03:54.840 --> 04:04.840 Yeah, but still, we still have infrastructures where we completely redesigned everything, 04:04.840 --> 04:14.840 and still we have popsary mail boxes or software or customers who use popsary, 04:15.560 --> 04:20.280 and it's sometimes only popsary, it's completely fine. 04:21.240 --> 04:27.400 Because, for like phone ISP, they sometimes have customers which have a contract 04:27.400 --> 04:37.240 over 20 to 25 years, and now when they say, okay, we want to drop popsary as a support, 04:37.240 --> 04:43.480 they were all called the company and saying, what's going on, what to do now, and they 04:43.480 --> 04:50.120 may be see that also the price, the prices are better at the companies currently. 04:52.440 --> 05:01.720 The other is about old software. You all know this SFTP is a transactional real-time protocol for 05:01.800 --> 05:14.280 exchanging data. Yeah, it's still the case, so we have some infrastructures where data 05:14.280 --> 05:25.080 is sent with SMPP, fetched with pop. Do you know the staff cards MD box and the feature 05:25.080 --> 05:30.440 that you have to purchase the mailbox from time to time? Do you know what happens when you don't 05:30.920 --> 05:38.120 purchase? You did it emails, but you don't purchase, so it will not be deleted from the 05:38.120 --> 05:45.720 MD box fire, and then what's happening when you forgot it and had millions of emails? 05:45.720 --> 05:52.840 And your monitoring is not up to date. Did we mention that this is critical infrastructure? 05:53.000 --> 06:03.400 Yeah, so we do see things that, yeah, we can't name names, but the systems are still out there. 06:05.240 --> 06:13.640 Yeah, another thing is about freedom of speech and the science, and here we are looking at 06:13.640 --> 06:22.040 the difficulty, for example, how to enforce the market SFTP policies. I have been sending my mail 06:22.040 --> 06:27.400 through Gmail all the time, so I need to do that. I'll have to send it out via Gmail or or or. 06:29.400 --> 06:37.480 So when we are coming to infrastructure and we're saying, okay, we can put SPS out there and we 06:37.480 --> 06:44.200 can do, and then suddenly we see how many actual clients are still not or would be rejected if 06:44.200 --> 06:51.240 this is actually then put into place hard. Yeah, these are day-to-day problems that we are 06:52.840 --> 07:01.800 often interfacing. The other thing is also on the inbound is that, oh, it is super, super 07:01.800 --> 07:06.680 important for me that I get all the email, but we have a little bit longer one, 07:06.680 --> 07:18.120 if we're kind of the topic of the chapter. Yeah, for this, we can always say there are 10 types 07:18.120 --> 07:27.560 of postmasters, also know what to do in those who do, but not really, it's always about resources 07:27.560 --> 07:34.840 about the power of other people, so if a local prof at the university has good connections to the 07:34.840 --> 07:42.040 port of directors and so on and that they're nice stories inside. There are also companies, 07:42.040 --> 07:48.120 also universities which are doing completely fine, was was all of the security stuff was D-Mark 07:48.120 --> 07:57.400 SFF and so on. There was a talk announced from the University of Bond, Peter Veneman, 07:57.400 --> 08:03.080 it's a pity he's ill, so that it was set was one of our very, very good projects, 08:04.040 --> 08:10.600 but there, as you can see, there are those where the prof still using his Gmail and sending 08:10.600 --> 08:22.760 was a university address. This is also one of the things, there seems to be a good agreement 08:22.760 --> 08:32.120 of postmasters who don't have enough resources to do it in a good way, so that they have to 08:32.120 --> 08:41.080 manage not only made but also storage and databases and maybe also the idea. And the funny thing 08:41.080 --> 08:51.800 is when there was a pushing wave inside the university, one or two weeks later, there's a 08:51.800 --> 08:59.160 maze bike beginning on Friday, 6 p.m. and ending on Monday, maybe 4 a.m. 09:02.520 --> 09:10.600 Universities are often very good in targets and infrastructures to sending pushing 09:10.600 --> 09:24.520 teams, because there's always has perfect resources network connection and so on. 09:24.840 --> 09:29.800 So yeah, for the last one also, you'll come back on Monday and you see you're on the outbound 09:29.800 --> 09:33.560 servers, there's a mail queue of several hundred thousand, you'll know something went wrong. 09:34.440 --> 09:41.480 The next thing is our real-world experience and to end encryption. So we're very much for it 09:41.480 --> 09:48.680 and we really love it and we would love everybody to include PGP and so on. In big corporations or 09:48.680 --> 09:54.920 bigger establishments, it becomes very difficult for the management of all those certificates. 09:54.920 --> 10:03.880 So what we see is that if you want to have it end encrypted, you can also say, okay, 10:03.880 --> 10:09.640 if we can ensure that every part is actually TLS encrypted on the way, we have basically an 10:09.640 --> 10:16.840 end-to-end encryption, but that is not sufficient according to some German official policies, 10:16.840 --> 10:21.480 yeah, that is okay, okay, then of some interpretations of German policies, this is not sufficient, 10:22.120 --> 10:30.040 sorry, I'm stand corrected. And so what we need to then do is to enforce this is to implement 10:30.760 --> 10:40.840 S-mime gateway. So ideally it would look like like the first base, but now so the problem is that 10:41.400 --> 10:47.240 you cannot have the users every one their own certificates, so you use something like a domain 10:47.320 --> 10:52.760 certificate and the user sends to the MTA, the MTA sends it to the encryption gateway, the encryption 10:52.760 --> 10:57.960 aches it back to the MTA, then it leaves the site and on the other side you have the same thing 10:57.960 --> 11:05.000 and goes to the user. Would you call this end-to-end encryption? No, thank you. But there's 11:05.000 --> 11:10.840 are the crimes we have to deal with in the real world and so this is what then is being built. 11:11.560 --> 11:20.520 And then sometimes some of these gateways are not multi-tenant able and then you have within the same 11:20.520 --> 11:26.360 organization is from one client to send to the other and then it does that, yeah, so the user sends to the 11:26.360 --> 11:31.480 MTA, the MTA to the encryption gateway, encryption back to the MTA to this heck hand encryption gateway because 11:31.480 --> 11:38.440 that's for that domain and sends it back to the MTA and then to the second user. And did I mention that 11:39.400 --> 11:49.480 we are not able to always enforce TLS internally? Okay, yeah, so sometimes you just wonder why 11:49.480 --> 11:55.320 things are the way they are, but again you have to deal with the realities of life. 11:58.440 --> 12:05.320 Yeah, but then it's how it's meant to be, so it was enter entered the encrypted in between 12:05.560 --> 12:17.880 for two seconds, maybe. Yeah, another thing, the fear of missing email, not formal for me, 12:19.960 --> 12:29.960 this is sometimes we redeploy complete infrastructures and then we get those questions 12:30.920 --> 12:36.600 what will happen to an email? I don't want to send bounces back, it looks bad to the send 12:36.600 --> 12:44.600 I've heard about if I send bounces back. What if if I want this email and it was rejected, 12:44.600 --> 12:53.880 what should I do? I want it. And sometimes it's a bit like this last question, what if my 12:54.840 --> 13:01.160 connection will break and then I don't have this email locally? It sounds a bit like it is 13:01.160 --> 13:13.480 from this time here, so it's really even today we are sometimes we sometimes still do things where 13:14.360 --> 13:21.800 emails are discarded, where they're ground time or something. We don't love ground time, 13:21.800 --> 13:31.480 we hate this card at this point and we always would, would favour reject everywhere, but sometimes 13:32.440 --> 13:44.520 it customers says I need it, please please do so. Yeah, also a nice thing, we have protocols for 13:46.440 --> 13:53.880 for email authentication. We all know that they have their problems, 13:54.760 --> 14:03.640 with SPF was forwarding and everything, but is it possible to enforce those? 14:06.280 --> 14:16.200 Yes, it's possible. No, no, it's not. If I've worked in the company, was it say it's team? 14:16.520 --> 14:24.440 Yes. No, it's time for the experience that someone from the state teams came out. Okay, 14:24.440 --> 14:32.360 this got this email back. Why have we talked to me? Yeah, our SPF was, this email was forwarded and 14:32.360 --> 14:39.480 our SPF was not correct. Well, and the guy said, I don't care. It makes that it works. Put 14:40.360 --> 14:48.200 of the records. There are always those two types. One of them says they want to have 14:48.200 --> 14:53.720 every email. They have those forwarders, they have those email addresses, they have their, 14:55.400 --> 15:04.520 not only WordPress, but let's say in the first structure, which is sending out emails in their 15:04.520 --> 15:12.280 name, and they still want to get them. Yeah, and the other type, best, best, since you have 15:12.280 --> 15:18.760 in the, on the same server, they say, okay, I really want those policies applied because it's 15:18.760 --> 15:27.720 part of the May security, and we really would love to do so, but it's not possible. Often, 15:27.720 --> 15:35.000 it's not possible. You can do per user. Let's say it would be perfect. 15:37.000 --> 15:45.320 Yeah, ARC is awesome, and I wish everyone would employ it, and would employ it well. 15:47.320 --> 15:56.040 Yes, yes, yes, yes, yes. Perfect. All right, all right, we're waiting for it, we're waiting for it. 15:57.000 --> 16:02.360 But at least it helps us for all these forwarding stuff, but then you have these big pairs, 16:02.360 --> 16:10.680 and you're really surprised. This is just from the logs of a couple of days ago, and one would 16:10.680 --> 16:19.800 assume they can do it. So they sign and then they change the mail. And this is not the one off. 16:19.960 --> 16:28.840 Yeah, so yeah, we're, we're sometimes just surprised. And yeah, I think, 16:35.240 --> 16:39.240 yeah, without words. This one for you. 16:39.400 --> 16:54.360 This is mine. Yeah. What to tell here? Yeah, it's only what we said on others' slides. 16:54.360 --> 17:05.640 We have, with Postmasters, we have, at least they like to do the old way, and when we 17:06.120 --> 17:15.880 deploy the infrastructure, we nearly do the same infrastructure as I had before. So working with 17:17.000 --> 17:26.920 files everywhere. Yeah, staying was, was handwritten, whitelist, blacklist, and everything. 17:27.080 --> 17:38.120 And yeah, it's, it's a bit of a generation change list, let's say, to, to bring new 17:38.840 --> 17:46.360 storages. So there are black boxes now. So S3, things like that, when you're talking about 17:47.080 --> 17:50.680 foundation to be Cassandra or everything, it's not a file anymore. 17:56.920 --> 18:05.800 Can maybe cut this short. Yeah, we also tried AI was, it took, it took sometimes, 18:05.800 --> 18:11.560 sometimes what faster, sometimes. Yeah, that's some emojis. 18:13.720 --> 18:21.400 Yeah, so there's other thing that we've come into that, so before we were often 18:21.400 --> 18:25.640 helping admins on their local setup, and we just kind of came in, did some consulting, 18:25.640 --> 18:32.600 and then they run it. And there's a whole kind of team, and we just kind of do a little part with 18:32.600 --> 18:39.880 mail. But more and more, we are now tasked to set up the whole infrastructure. Not just the 18:41.480 --> 18:49.080 the multifactor authentication, but with everything, with, yeah. So here, you have your 18:49.480 --> 18:59.800 ODC connection, log on, that's your resources, make a work. And yeah, this is the infrastructure 18:59.800 --> 19:07.800 that we deploy in, in our scale, and these are where all the tools that many of you are writing, 19:07.800 --> 19:15.960 and providing that we are kind of putting together to cover the needs of these 19:16.920 --> 19:22.360 institutes. And so these are not just some tools, but they are actual things. 19:22.360 --> 19:26.920 So we're really looking forward to start with as one of the center, who has seen our talks, 19:26.920 --> 19:36.920 knows that we're working a lot with ARSBIMD, and as the for us, the core for all of the security 19:36.920 --> 19:42.840 needs. And we're looking forward to implementing, not to this, that is, I'm being developed. 19:43.560 --> 19:50.360 And yeah, for ARSBIMD, why we still think that that's such an important part in this, 19:50.360 --> 19:57.080 because it has the possibility to actually orchestrate a lot of the other tools that are 19:57.080 --> 20:05.080 rounded from file analysis, sandboxing to the antivirus, and we can't even maybe have an 20:05.080 --> 20:11.640 export to seem. So all of these things are actually being asked. So that's why we always look 20:11.720 --> 20:15.960 nicer when you just have one server kind of thing, and that takes care of my email, but in the end, 20:15.960 --> 20:24.360 we all know at one scale that won't do it anymore. And what we really would love is that the 20:24.360 --> 20:29.240 outgoing internet proxy can also then inform us about when the link was clicked that we saw earlier, 20:29.240 --> 20:34.680 and if it was good or bad. So then afterwards the value of the email. But we're not there yet. 20:34.680 --> 20:44.600 Yeah, you want to intro? Yeah, not to lose is one of those components. And so we have seen the 20:44.600 --> 20:53.160 the need for an IDP and for brute force. And so we're seeing this project coming up. And it 20:53.160 --> 20:59.160 really really looks promising. We have not integrated it yet in our production infrastructure, 21:00.120 --> 21:08.040 but yeah, do check it out. It's really, really, really, really interesting. And yeah, we we see this 21:08.040 --> 21:18.520 as the other core component that will be part of that. And yeah, it does a lot from brute force, 21:18.520 --> 21:25.160 real-time RBL's network security, and has these custom API endpoints, and as well. Now open 21:25.160 --> 21:29.560 telemetry support. And I think, in the next version, I talked to the developer, 21:30.360 --> 21:36.760 they want to also have IDP as an IDP, so that you can also do authentication against it. So that's 21:36.760 --> 21:43.160 fun and interesting. And we're looking forward. Yeah, here. 21:43.400 --> 21:55.160 Is that mine? Yeah, we've talked about the email security and the whole infrastructure. 21:55.160 --> 22:01.160 So if you want to have a big enterprise and everything, you can pay hundreds of thousands of 22:01.160 --> 22:08.600 dollars to get one of those. We know Sandboxes to from one of the big 22:08.600 --> 22:17.000 antivirus supplies. There's also an open source community project there. They're nice. 22:17.880 --> 22:22.120 Sandboxing tools. Pick a boo-a-v is a like a wrapper or 22:23.480 --> 22:28.280 evaluation service for all the outputs of those Sandboxes. And we have integrated it into 22:28.280 --> 22:34.760 us from B. So it's not an official plug-in, it's a bit under the line still, but it runs 22:34.920 --> 22:44.440 productive for several years at a project. So we also have to give it some time. We have 22:44.440 --> 22:51.160 rejected the email waited five minutes, so that's the Sandboxing was fine. We have this pre-precure 22:51.160 --> 23:00.120 everything. If you'd like to have something like Sandboxing included, they definitely search for 23:00.760 --> 23:08.200 some new project. There's some challenging things mostly on the side of the Sandboxing. 23:08.760 --> 23:14.680 Because Sandboxing is a problem that those tools like Kuku are only capable to do 23:15.240 --> 23:22.440 Windows 7 and all the following one. Yeah, needs still some development. 23:24.760 --> 23:26.760 Thank you. 23:30.120 --> 23:44.440 Perfect. Thank you.