WEBVTT 00:00.000 --> 00:12.080 Alright. Good morning. Welcome to the Package Management Devroom. Thanks Andrew and 00:12.080 --> 00:17.240 Wolf for having me. I'm Adam, as it says on the screen. I'm here in a couple of capacities 00:17.240 --> 00:22.240 today, both as a staff member of the Rust Foundation, where I'm employed as a security focus 00:22.240 --> 00:26.680 software developer. So if somebody steals your AWS credentials through some of the 00:26.680 --> 00:31.720 great-born malware, that's probably my fault. And I'm also here as a member of the 00:31.720 --> 00:37.160 crates.io team. And for those of you who don't know, crates.io is the Package Registry for 00:37.160 --> 00:41.960 Rust, basically. It's the official Package Registry. Same as PIPI, or MPM, or other similar 00:41.960 --> 00:48.840 pictures. So since, though, a nice enough to put me up first, we're going to sit around a 00:48.840 --> 00:52.800 very virtual campfire because of fire regulations and talk about something that happened 00:52.800 --> 00:58.400 last year. I thought about getting marshmallows, but it just didn't seem like it was worth it. 00:58.400 --> 01:03.440 So as I've mentioned, Crates.io is the official Package Repository for Rust. Hosts 01:03.440 --> 01:07.760 about 220,000 crates, which is the Rust term for packages, because of course we have to invent 01:07.760 --> 01:14.520 our own word. And is growing rapidly, much as Rust is in general across many things. 01:14.520 --> 01:20.160 On a technical level, Crates.io uses GitHub as its sole identity provider. This was a decision 01:20.160 --> 01:25.520 it was made back in 2014 when Crates.io was first created. It made sense at the time. There are 01:25.520 --> 01:30.240 many good reasons to support other identity providers in the air of our ward 2026, but that 01:30.240 --> 01:35.280 hasn't happened yet. It's an active area of discussion. If you would like to discuss that further, 01:35.280 --> 01:39.440 Toby is somewhere over there, who is the team lead. He will absolutely hate me, don't think 01:39.440 --> 01:45.840 him in. But it's so pretty well overall. The big benefit we've had is we've got to benefit 01:45.920 --> 01:50.560 from GitHub needing to take active measures to protect their own platform, to control spam, 01:50.560 --> 01:56.080 to prevent fishing attacks. And it's meant that we haven't had to think about policies around things 01:56.080 --> 02:03.360 like account recovery, succession matters. The only real downside apart from freedom has been 02:03.360 --> 02:08.160 the, you know, mild burning sensation of being reliant on a large American company, which is 02:08.160 --> 02:13.440 a little bit less fun than it used to be. The growing popularity of Rust, though, has made it 02:13.440 --> 02:18.080 a more attractive target. And there are some downsides to having a single idea as to provide 02:18.080 --> 02:22.480 her as well. One of the big ones is you don't have to worry about standing up multiple fake 02:22.480 --> 02:27.760 authentication flows if you're trying to fish once a fishing attack. You just have one and 02:27.760 --> 02:33.520 everybody knows what GitHub author looks like, right? We probably all click through it like multiple 02:33.520 --> 02:39.440 times a week for different things. And Crates, although I follow, is a pretty normal 02:39.440 --> 02:43.760 power law distribution when it comes to package or create ownership. If you can just pop one of 02:43.760 --> 02:48.320 the accounts of the top, you know, 50 or so crad owners, you go and control over millions upon 02:48.320 --> 02:52.880 millions of downloads per day. Many of those are going to CI, but some of those are going to 02:52.880 --> 02:58.880 develop a machines and developers tend to have poor security hygiene. So that's a good, that's a 02:58.880 --> 03:03.840 lot of potential asternaticans. Now, of course, we're not the only ones in this boat, 03:03.840 --> 03:09.200 NPM and PIPI, not among others, see at least an order of magnitude, more packaged downloads and 03:09.200 --> 03:14.960 Crates.io. And, you know, while Rust is still making inroads in some places, JavaScript, 03:14.960 --> 03:23.520 TypeScript, and Python are used by everyone. So last year, there were some fishing. People 03:23.520 --> 03:30.240 try to do bad things. The first target was actually NPM. In mid July last year, a developer who owned 03:30.320 --> 03:35.760 several popular lending packages got targeted by an email that actually was this, and generated an 03:35.760 --> 03:40.720 API token that could then be used to publish new versions of one of their packages that contained 03:40.720 --> 03:46.080 malware that's currently being called scavenger. If you look, it's going to be a little bit hard to 03:46.080 --> 03:53.040 read, but basically the link here goes to NPM for November.js.com. So that's reasonably sneaky, 03:53.040 --> 03:56.240 right? Like, if you're a half asleep and you're just looking for your emails in the morning, 03:56.800 --> 04:02.640 I probably wouldn't have spotted it. But truthfully, this campaign really first came to my 04:02.640 --> 04:08.720 attention when this blog post has published on the Python blog, PIPI blog a couple of weeks later 04:08.720 --> 04:12.240 by my friend Mike, who works at PIPI and the Python software foundation. 04:13.280 --> 04:17.760 PIPI had seen a fishing campaign about a week after the NPM attack, where the attacker was trying 04:17.760 --> 04:24.480 to harvest credentials via the PIPJ.org domain. So in a sense of theme here, and had in fact succeeded 04:24.480 --> 04:28.720 at fishing four user accounts, including the owner of a reasonably popular Python package, 04:28.720 --> 04:34.480 which was about 1600 in the download charts, which considering this over 700,000 packages on 04:34.480 --> 04:40.240 PIPI is pretty good. A new version that package was published with, as I understand, it's suspiciously 04:40.240 --> 04:46.800 similar malware, and again, an API token was safe for further use. So this kind of a pattern here, 04:46.800 --> 04:52.480 right? Like, you know, we've got some pretty cleverly named domains. We've got some emails. I don't 04:52.560 --> 04:57.520 have a screenshot of the email. I don't think I don't think PIPI had one, but presumably looked 04:57.520 --> 05:01.440 reasonably authentic, much as the previous one there did. That looks, you know, not bad. 05:02.960 --> 05:08.400 And they obviously built a good facsimile of the login experience, and they're not completely 05:08.400 --> 05:13.360 idiots about this, because they're stashing API tokens for later. The PIPI one actually published 05:13.360 --> 05:18.000 a second version of the package after the first version got taken down with the same malware. So 05:18.080 --> 05:23.280 they had an API token, and it wasn't until the author went and looked at their token settings, 05:23.280 --> 05:30.080 that they actually figured out that they still had a token. So we fast forward into early September, 05:30.080 --> 05:35.280 there was another instance of this, very similar pattern, which can compromise these packages 05:35.280 --> 05:40.320 in the MPM world, chalk and debug. I'm not going to go into any great detail on this one, but 05:40.320 --> 05:44.400 again, you can kind of see this. The other thing that was interesting about this was, 05:45.360 --> 05:50.240 most fishing attacks against registries historically, and I'm indebted to my colleague Walter 05:50.240 --> 05:56.320 who also works at the Rust Foundation for this. Kind of a spray and prior approach, right? Like 05:56.320 --> 06:00.480 if you could get an email for somebody who had an account on a registry, they would just send 06:01.120 --> 06:06.480 the email to everyone. These attacks were a bit more targeted. Like they were just targeting people who 06:06.480 --> 06:14.320 had high end packages, you know, people who were owners of, you know, top 1% packages in their 06:14.320 --> 06:19.840 respective ecosystems. So there's a bit more effort being put in. All right, so this was a Monday, 06:19.840 --> 06:26.320 September the 8th was a Monday. Let's go forward to Friday. So this discussion is open, 06:26.320 --> 06:30.560 it's a little bit carefully edges, but I'll talk through on the crates.io GitHub repo. 06:31.200 --> 06:39.040 Basically, it is, as it says at the top, crates.io fishing, and somebody had received a fairly, 06:39.040 --> 06:44.720 fairly decent looking email that basically said that crates.io had been compromised, and they had 06:44.720 --> 06:49.680 access some user information. We are currently drafting a blog post to outline the timeline, 06:49.680 --> 06:55.120 and we suggest that you rotate your login info by signing into our internal SSO, which is a temporary 06:55.200 --> 07:00.000 fix to enjoy your attack account, modify any packages published by you. Now, you'll note something 07:00.000 --> 07:05.840 right away in the screenshot, which is, this goes to github.rustfoundation.dev. My assumption here is that 07:05.840 --> 07:11.120 they couldn't come up with a sufficiently confusing version of the crates.io domain name, which really 07:11.120 --> 07:17.200 just reveals a lack of imagination, I think. But regardless, they then attempted to use the good 07:17.200 --> 07:22.880 name of the rust foundation to basically say, it's okay, we set up our own SSO coming here. 07:23.840 --> 07:30.960 Toby, again, sitting over there, saw this in his notifications and sent a message to the, we have a 07:31.760 --> 07:37.040 two, a channel on Zoom up. So Zoom up is what the rust project uses for communication, and we have 07:37.040 --> 07:41.680 basically security responses handled by the rust project has a security response working group, 07:41.680 --> 07:45.600 and then they handled them in collaboration with whatever the effect the team is. So in this case, 07:45.600 --> 07:50.080 the crates.io team. Toby sent the message to that a bunch of people's phones started buzzing, 07:50.080 --> 07:54.480 including mine, because we have notifications turned on for that stuff, and away we went. 07:55.200 --> 07:59.600 We also had about the same time, got additional confirmation from others that they were seeing the same 07:59.600 --> 08:04.720 sort of thing. So this is, this is, for an expert sushi, he was a very prolific crates author in the 08:04.720 --> 08:11.200 rust ecosystem. So I'm going to talk about the response, I'm going to avoid using any names 08:11.200 --> 08:16.240 other than Toby's for the rest of this. Basically, just because every involved degree work, 08:16.240 --> 08:21.280 I'm happy to give them public shout out if they want them. But this was a fast-moving private response 08:21.280 --> 08:24.960 thread. I'm not going to put screenshots on top of like rural conversations here because it 08:24.960 --> 08:28.960 moved fairly fast. I don't want to attribute things to people without full context or permission. 08:28.960 --> 08:32.960 So I'm going to say we a lot, but in most cases it's going to be an individual. 08:34.160 --> 08:40.880 So at this point, we collectively, plural, spring into action. Personally, I was in the back 08:40.880 --> 08:46.320 seat of a DD in Hangzhou, China, for a conference dinner. So I was a long way from my laptop, 08:46.320 --> 08:50.480 which was really annoying. But there were many others who were thankfully at their laptops, 08:50.480 --> 08:55.440 or at least closer laptops and could start working right away. Initially, we had two main areas of 08:55.440 --> 09:00.720 focus. One, communicating there was an active attack that we're aware of it, and reiterate the 09:00.720 --> 09:05.200 crates author is never going to send you an email asking you to log in to something else, or really 09:05.200 --> 09:10.720 anything ever, including crates.io. And two, start working to take that Rust Foundation.dev 09:10.720 --> 09:16.320 domain name down. You know, the Rust Foundation, somewhat infamously has trademarks. 09:17.600 --> 09:21.600 And you know, it felt like an opportunity to actually use them for something useful rather than just 09:21.600 --> 09:28.400 scaring people with trademark policies. So as a side by just on that, I've kind of actually really 09:28.400 --> 09:32.480 a little bit tickled that they use the Rust Foundation's name because the presentation doesn't 09:32.560 --> 09:36.560 technically run crates.io, or they're fun to the infrastructure, doesn't own Rust. 09:37.440 --> 09:42.320 And has a really mixed reputation in the Rust community. So invoking the name of the Rust Foundation, 09:42.320 --> 09:47.440 there's to make it look authoritative, was definitely a choice. Maybe not a very good one. 09:48.320 --> 09:52.800 The main Rust Foundation.dev domain, by the way. So there's a screenshot of the GitHub sub-domain, 09:52.800 --> 09:57.440 where I had like a nice little offflow. The main Rust Foundation.dev domain at this point 09:57.440 --> 10:02.640 contained a recoil. It was just like an embed of rigorously. Following that domain, we're going 10:02.640 --> 10:06.400 to come back to that. Following that domain ended up being kind of fun. It was kind of a communication 10:06.400 --> 10:14.240 channel through this process. So one of the things that we was sort of trying to figure out in 10:14.240 --> 10:20.880 the early stages was, so at what point do we talk to GitHub and how do we talk to GitHub? Because 10:20.880 --> 10:25.440 this fishing attack is going, obviously, being is going through GitHub infrastructure at some point, 10:25.440 --> 10:28.960 right? Like they've got the fake part of logging flow. But these are eventually going to 10:28.960 --> 10:34.160 turn into a real request to GitHub API. They're going to sign in. They're going to then get a 10:34.160 --> 10:38.800 session token that they can exchange for creates.io or then presumably create a token as 10:38.800 --> 10:44.960 what we assume is going to happen. The problem with that is, where do you report that to GitHub? 10:45.760 --> 10:52.000 How do you report that GitHub? Now GitHub has lots of report buttons on repos and things like that. 10:52.640 --> 11:00.800 This isn't tied to a repo. In the end, we sort of dug up an email address from I think a 11:00.800 --> 11:07.120 who is record and went with that. This was not super successful, as it turns out. 11:08.320 --> 11:12.160 So we're going to come back to this. I'm going to talk more about the party provider 11:12.160 --> 11:16.880 swars in the end. But just kind of keep them in the vacuum mind that we were, you don't like having 11:16.880 --> 11:20.640 uncertainty when you're responding to something and this was a significant source of uncertainty. 11:22.400 --> 11:26.320 We also discussed whether we'd be able to detect anything from the crates.io 11:26.320 --> 11:30.640 observability stack that we have. We realized pretty quickly that it was very little we can do 11:30.640 --> 11:35.680 directly because the fishing attack wasn't actually touching crates.io in any way. They probably 11:35.680 --> 11:43.280 weren't going to make a request to crates.io API server until they had a login, a fish login 11:43.280 --> 11:49.120 through GitHub. There was no reason otherwise to touch the API. That made monitoring this kind of 11:49.120 --> 11:54.160 difficult. We didn't really know if somebody had actually gotten fished or not. We do have the 11:54.160 --> 11:58.160 ability to put crates so into read-only mode, but we agreed that doing so at that point felt 11:58.160 --> 12:02.400 fairly premature. We didn't have a lot of evidence as being a widespread campaign and 12:02.400 --> 12:07.680 read-only mode involves a lot more disruption than just log-ins, like people can't publish crates. 12:07.680 --> 12:14.880 They can't. There are many things they can't do. While this was going on, part of the group 12:14.880 --> 12:18.560 were able to get a initial blog post and some social media posts up, noting that the attack was 12:18.560 --> 12:23.280 going on, and read-writing that crates would never actually send you an email asking you to log in 12:23.280 --> 12:27.760 somewhere. This is about an hour into the response, and at this point we're kind of in a holding 12:27.760 --> 12:32.640 pattern. GitHub hadn't gotten back to us, nor had the domain register, nor had the host, 12:32.640 --> 12:38.400 the russfoundation.dev was being hosted on. We did some ad hoc brainstorming on Zoom up about 12:38.400 --> 12:43.040 things we could do differently. I will talk about a couple of those tools again, but basically 12:43.040 --> 12:47.920 we're sort of in a holding pattern at this point. That and hour after that, so about two hours 12:47.920 --> 12:52.080 after the start of the campaign, the russfoundation.dev website changed, so it was no longer a 12:52.080 --> 12:59.920 recoil. It looked like this. First things first, I don't actually think russ to help as a 12:59.920 --> 13:06.000 smart and then JavaScript developers. Maybe better looking, but give us that. But it did give us 13:06.000 --> 13:10.640 a laugh, so thank you, Mysterious Fisherpeople. They were not done, though. We'll come back to that. 13:11.600 --> 13:15.920 A suggestion was made in the public security response working group channel around the same time, 13:16.560 --> 13:21.040 was whether the russfoundation.dev domain should be submitted to the Google Safe Browsing Project. 13:21.920 --> 13:26.320 That honestly hadn't occurred to us, and I'd be interested to know to this dad, 13:26.320 --> 13:29.360 I don't really know if we should have done it or not. We didn't end, I don't think anything 13:29.360 --> 13:33.200 actually happened, but I'd be interested to know if that's a standard part of anyone else's 13:33.200 --> 13:36.880 protocol here. Because I didn't really think that we didn't really think of that. 13:37.840 --> 13:45.680 So that's pretty much the initial few hours. As far as we can tell, and I'll explain kind of why 13:45.680 --> 13:50.480 I think this a little bit later, nobody in the targeted set of users, and to this day we don't know 13:50.480 --> 13:55.440 how many that was or exactly who it was, we're deceived by the Fish and Campaign. 13:56.000 --> 14:00.640 Over the following day, also the russfoundation.dev website changed a couple of times. The next 14:00.640 --> 14:05.120 change was to this. I will read it just for the better for the people at the back, 14:05.120 --> 14:09.920 Crates.io database along with juicing tokens for sale, email for buying, free leak if 14:09.920 --> 14:14.000 no offer till Sunday. So I think this is on Saturday. Security at russfoundation.dev. 14:14.800 --> 14:20.800 Then that was followed by sold. Wow, that was quick. Who ever thinks this was sophisticated 14:20.800 --> 14:24.640 is clearly fear mungering. Less than 15 minutes, less than 30 minutes, et cetera. 14:26.400 --> 14:30.640 I'm going to say several months later with a fair bit of confidence that there was no compromise 14:30.640 --> 14:35.200 of the crates.io database. We're pretty sure this is a last gas piece of the domain 14:35.200 --> 14:40.480 trying to fish some cryptocurrency out of a sucker rather than create some fud rather than an 14:40.480 --> 14:45.360 actual breach. I'm also guessing that apart from the mean they weren't super happy with how this 14:45.360 --> 14:53.200 had all gone. So you know, sucks to be there. So through this process, one of the nice things was 14:53.760 --> 14:59.120 we had a lot of cooperation. A number of people have spent time in the last few years, including 14:59.200 --> 15:03.600 these two gentlemen, working to build a community of package registries and managers. 15:04.880 --> 15:08.160 Alfred Omega, and especially Michael Windsor, who is also giving a talk later, 15:09.120 --> 15:13.520 played a huge role in fostering connections between the language package registries and that's 15:13.520 --> 15:18.960 been incredibly helpful. So this was really clutching a couple of specific ways here. One was 15:18.960 --> 15:22.400 it's just nice to have a friendly sympathetic year. When you're going through this kind of thing, 15:22.400 --> 15:26.400 like being able to just go to like a signal group chat or a Slack channel and just be like, hey, 15:26.560 --> 15:30.720 this is happening and have a bunch of people go, yeah, that happened to us at sucks is kind of nice. 15:31.840 --> 15:37.120 But secondly, having a larger group of people means you have by extension, a larger group of 15:37.120 --> 15:42.560 contacts who you can get to. So you'll know that when I left off the story, nothing actually 15:42.560 --> 15:48.080 being taken down. So, you know, again, spoil us for kind of later. But that was a way that we were 15:48.080 --> 15:54.720 able to get more contact, we're at least some of the providers involved. But here's another concrete 15:54.720 --> 16:00.080 way that helped. Remember the PIPI attack that I mentioned. Seth Larson and of PIPI and the PIPI 16:00.080 --> 16:04.160 for Software Foundation had set up the main threat and monitoring of newly registered domains to 16:04.160 --> 16:08.800 try and find these things before the attacks actually launched. A couple of weeks after that 16:08.800 --> 16:13.600 Crate Solo attack, Seth was alerted to a domain that was likely to try to fish PIPI again. It was 16:13.600 --> 16:18.480 squatting some sort of typo for PIPI. But it turned out it was actually at that point, 16:18.480 --> 16:23.440 misconfigured to serve up a Crate Solo authentication flow on what was apparently going to be the 16:23.440 --> 16:30.400 prod 01-crate.io domain. So, let's take a little bit more creative. Now, we can't prove that was a 16:30.400 --> 16:37.200 same group as before, but it does feel kind of familiar, right? So, we spun the initial response 16:37.200 --> 16:42.000 thread back up and another member of the Crate Solo team created a new GitHub and Crate Solo account 16:42.000 --> 16:48.000 that had no connection to anything, no access to anything. And so, using that we were able to actually 16:48.000 --> 16:54.320 monitor what would have happened had somebody fallen for the fishing attack. Basically, a Crate 16:54.320 --> 16:58.960 Solo API token was registered with full scopes and a unique name. Like it had, it was things 16:58.960 --> 17:02.400 like underscore underscore underscore underscore something like that. Something that we didn't 17:02.400 --> 17:07.360 have any other examples of in the database. We were then able to find a bunch of tokens that had 17:07.360 --> 17:12.000 been registered 11 hours earlier by another new GitHub account that was presumably testing the 17:12.000 --> 17:16.160 flow, making sure that it worked, making sure they could create API tokens, making sure the payload 17:16.480 --> 17:22.080 worked. So, what that also meant was we now knew what to look for in the observability setup. 17:22.080 --> 17:27.520 So, we're able to go back to the previous attack and see whether anybody had, in fact, tried to 17:28.160 --> 17:32.960 try to actually, or had fallen for them. We had asked people to report if they had, if they had 17:32.960 --> 17:38.080 been fished and we did say there would be no shame, but people still going to feel shame. So, 17:38.080 --> 17:43.040 we didn't hear from anyone, but we didn't want to take those concrete proof. This strongly hinted 17:43.040 --> 17:47.440 that we were, in fact, like nobody in the Rust, but nobody in the Crateslayer community had, 17:47.440 --> 17:51.920 in fact, fallen for the fishing. So, we're able to do a retrospective query and confirm that. 17:52.720 --> 17:56.880 And with all of that, we now had enough, we had enough data that we could go get stuff taken 17:56.880 --> 18:01.600 down and take defensive actions before anything actually got going to foil this campaign. 18:01.600 --> 18:06.400 As far as we know, that time around, no emails ever got sent. So, that was a win. Like, 18:06.400 --> 18:12.160 the best fishing attack is one where the attack never happens. So, that's all the nice warm 18:13.040 --> 18:16.720 fuzzy bits. The Rust project worked really well together. The teams and working groups involved. 18:16.720 --> 18:22.160 The Rust Foundation security team. I provided support. The broader package registry 18:22.160 --> 18:27.120 and security communities, especially PIPI and our for Omega, provided really good support and back 18:27.120 --> 18:32.320 up for us, and tipped us off to the abortive second attempt. So, now, let's talk about saying 18:32.320 --> 18:38.320 it was not quite as good. Commercial, third party providers. There were quite a few providers in the 18:38.320 --> 18:42.080 mix here. I'm only a name one in my name, but if you catch me after some beers later, maybe I'll 18:42.080 --> 18:46.320 name some of the others. The one on my name is obviously GitHub because they have to be involved 18:46.320 --> 18:52.480 they're the ideas to provide. So, when we went to GitHub in the first place, as I said, 18:52.480 --> 18:57.440 we emailed an email address, we pulled out of who is because we just had no idea who else to contact, 18:57.440 --> 19:03.680 didn't get a response. Eventually, we were able to get a lot higher level contact direct to someone 19:03.680 --> 19:08.560 at GitHub. This is several hours after the attack started via a colleague in Alpha Omega, 19:08.560 --> 19:12.640 and at that point, all of a sudden, the wheels started turning, unsurprisingly. And, you know, 19:14.000 --> 19:18.240 sort of, you know, put some monitoring in, tried to prevent authentication through that flow 19:18.240 --> 19:23.040 and so on and so forth. So, that's good. That's, you know, like, perfection is too much to ask. 19:23.040 --> 19:27.520 Honestly, once we had the right contact, that was great. It's a little unfortunate that we 19:27.520 --> 19:32.720 had to wait for that right contact, but, you know, that is what it is. There are also two hosting 19:32.720 --> 19:36.800 providers involved, both of whom were presumably chosen for their acceptance of cryptocurrency, 19:36.800 --> 19:43.600 and general just air of sketchiness, two domain name providers, and in the second attempt in 19:43.600 --> 19:47.040 campaign, a well-known CDN company, and yes, it's probably the one you're thinking of. 19:48.480 --> 19:52.560 The CDN provider took down the site in a pretty good turnaround times. About two hours, that's 19:52.560 --> 19:55.120 fair enough. We don't pay them. Like, that's, that's pretty reasonable. 19:55.200 --> 20:03.680 Um, nobody else ever responded. We sent emails. We sent social media, like, we tried. We really 20:03.680 --> 20:08.480 did try. I think, I think Wall Street even actually called someone on the phone, which is terrifying 20:08.480 --> 20:15.360 from a millennial. Um, and just nothing. Never heard back. Nothing ever got taken down, and that's 20:15.360 --> 20:19.840 really just disappointing. So, you know, that's kind of a cautionary tale that you are, 20:20.880 --> 20:24.720 as much as you can cooperate with other people, you may not get much cooperation from 20:24.720 --> 20:29.760 commercial providers. So, if that's a no and no, and then you can factor that into your 20:29.760 --> 20:35.040 internet response, right? We did make a couple of little changes out of this. Uh, Toby opened 20:35.040 --> 20:40.080 a PR to disable just API token creation rather than going to full read only modes. That's a useful 20:40.080 --> 20:45.600 thing. Toby looks really surprised. I think he forgot he did that. Um, that's really helpful. 20:45.600 --> 20:49.360 There's still some downstream effects, trusted publishing doesn't work, but it's better than nothing. 20:50.400 --> 20:53.760 Another thing we realized was that the rust security response working group didn't actually 20:53.840 --> 21:00.480 have a way of paging the on-call stuff for crates.io. So, we fixed that. Um, in this case it worked out 21:00.480 --> 21:04.000 because the attack was during the European Workday, the North Americans were just waking up, 21:04.000 --> 21:08.560 and even me in East Asia was still awake. So, everybody was basically up and in the loop. 21:09.360 --> 21:13.600 But we fixed that. So, that's a real lesson that we learned out of that. Make sure people can 21:13.600 --> 21:20.320 wake the right people up when this kind of thing gets noticed. But the main lesson really is 21:20.400 --> 21:24.480 we're better working together. We're more likely to get the outcomes that we need when we do so. 21:24.480 --> 21:28.880 So, with that, I'm glad that we're all in the room. Make sure you make friends with people 21:28.880 --> 21:32.720 that you don't already know. Um, and let's have a great day. Thank you very much. 21:38.720 --> 21:43.680 I'm going to start. I don't know if we actually have time for questions. So, that's okay. 21:43.840 --> 21:48.480 Do you have one question, but you can have to. If anybody has I'm burning, go for it. 21:50.480 --> 21:51.040 Yes, sorry. 21:56.960 --> 22:01.200 Okay, so the question is the original fishing attacks. How did they circumvent the two-factor authentication? 22:02.160 --> 22:05.680 That is an excellent question that I don't actually fully know the answer to. 22:06.720 --> 22:11.200 As far as I know, nobody did that level of analysis on the on the fake size that they were 22:11.520 --> 22:16.720 putting up to front them. They obviously were doing something to act as a proxy between the user 22:16.720 --> 22:22.480 and the actual and the actual identity provider. If we never got far enough into this to figure that out 22:22.480 --> 22:27.200 to be honest with you, if somebody MPM or PIP did that analysis, it's not in the blog post and I 22:27.200 --> 22:32.880 don't have any further details. I'm sorry. But yes, absolutely. That is absolutely a problem. 22:32.880 --> 22:37.840 And I wish I had a better answer. All right. Thank you very much. And enjoy your day.