WEBVTT 00:00.000 --> 00:08.880 All right, okay, let's go. 00:08.880 --> 00:18.040 So, I want to make the case for non-blocking code reviews as opposed to the classic commonly 00:18.040 --> 00:21.840 accepted pull request model. 00:21.840 --> 00:26.560 Just to be clear, I don't question the pull request model in the open source world. 00:26.560 --> 00:31.600 It has value for the open source world because it solves the trust issue. 00:31.600 --> 00:38.600 I do question it in the corporate environment where I think it has no value at all. 00:38.600 --> 00:49.760 Anyway, so back in 2012 quite a long time ago, so it's only recently I started speaking 00:49.800 --> 00:51.640 about this. 00:51.640 --> 01:01.120 I had the opportunity to start a coaching mission to upscale the engineering skills of a 01:01.120 --> 01:04.960 role in another team. 01:04.960 --> 01:07.160 Luckily, I wasn't alone doing that. 01:07.160 --> 01:14.960 We were two coaches and was my friend, Martin van Aiken and myself, which makes things 01:14.960 --> 01:16.960 more easy. 01:16.960 --> 01:22.000 And so when we arrived with this corporate situation, we didn't quite expect to still 01:22.000 --> 01:28.000 discover in 2012, no version control was in place. 01:28.000 --> 01:33.040 Yeah, so first thing we did was introduce a certain control system, and I thought like, yeah, 01:33.040 --> 01:40.400 well, get this maybe a better bridge too far for a team that never used any version control 01:40.400 --> 01:44.360 system while the whole idea of having a remote-reap or a local repo, when you commit it's 01:44.360 --> 01:47.560 not in the remote repo, it was like mind blowing. 01:47.560 --> 01:50.880 So I suggested we'll go for a version. 01:50.880 --> 01:57.160 And because it was said that while branching and subversion is difficult, complicated, I 01:57.160 --> 02:00.880 also suggested we use no branches at all. 02:00.880 --> 02:08.160 Everyone is going to commit straight into the main line, well, strength for a supership. 02:08.160 --> 02:13.600 And that went pretty well because well, the second thing we introduced right from the start, 02:13.600 --> 02:20.800 and that was the practice of continuous integration together with team commitment that any 02:20.800 --> 02:26.200 change had to be covered by not made the test before the unit test. 02:26.200 --> 02:31.880 Now because the team was nervous, well, we wanted to have some mentoring in place. 02:31.880 --> 02:35.600 And bare programming was difficult. 02:35.600 --> 02:42.560 The team was not particularly open to the idea, although we did some pairing from a coaching 02:42.560 --> 02:45.720 perspective, but it wasn't part of the team culture. 02:45.720 --> 02:49.320 So the next best thing was, well, good reviews. 02:49.320 --> 02:55.320 And back then, I was a fervent believer of good reviews, mainly influenced by books like 02:55.320 --> 03:01.320 good complete, facts and fantasies of software engineering and the writings of Joel Walspozki. 03:01.320 --> 03:03.720 Anyone remembers Joel Spalski? 03:04.280 --> 03:16.360 Yeah, well, it says something about our age, but anyway, and because the team was nervous, 03:16.360 --> 03:21.720 we also decided every single commit had to be reviewed. 03:21.720 --> 03:26.760 And so because there were no branches in place while we ended up doing non-blocking code reviews, 03:26.760 --> 03:31.960 which was, well, the realization of implementing lean principles and keeping things simple 03:32.040 --> 03:38.720 stupid, not really realizing that what we were doing was actually a performant way of running 03:38.720 --> 03:39.720 code reviews. 03:39.720 --> 03:44.840 And in all honesty, I was quite embarrassed by our way of working compared to the rest 03:44.840 --> 03:45.840 of the industry. 03:45.840 --> 03:50.520 So we were nowhere close to what the rest of the industry was doing at that time, using 03:50.520 --> 03:55.080 branches and code reviews and public has the platform. 03:55.080 --> 04:00.400 Now, there's non-blocking code reviews were based on two key ideas. 04:00.400 --> 04:05.920 And the first key idea was to establish a continuous review process that consists of regular 04:05.920 --> 04:09.240 reviews as part of the development process. 04:09.240 --> 04:13.520 So the team used a can-manboard to track work. 04:13.520 --> 04:17.560 And every ticket on the board was a feature, not a task. 04:17.560 --> 04:23.120 And so if you want to move a ticket quickly from left to right, well, it puts pressure 04:23.120 --> 04:26.440 to keep those features small. 04:26.440 --> 04:30.720 So the feature was a unit of work on the can-manboard, but the feature was not the unit 04:30.720 --> 04:33.760 of integration as it happens with feature branch. 04:33.760 --> 04:36.200 So the unit of integration was a commit. 04:36.200 --> 04:42.120 And the unit of release and deployment into production was a commit or a series of commits. 04:42.120 --> 04:46.800 Because the decision to deploy into production was still a manual decision, we practiced 04:46.800 --> 04:49.320 continuous delivery and not continuous deployment. 04:49.880 --> 04:56.040 Now to ensure that code reviews happened and to track the code reviews, we introduced a 04:56.040 --> 05:01.840 to review column, which indicates this feature is ready to be reviewed. 05:01.840 --> 05:05.120 So it's good complete, ready to be reviewed. 05:05.120 --> 05:11.480 Now, Dave Farley has seen this before, but it never really worked out for the teams 05:11.480 --> 05:15.720 that Dave worked with because it was always seen as a gate. 05:15.720 --> 05:23.280 And research is clear about process gates, so going slow for safety is a bad idea, 05:23.280 --> 05:30.400 because it introduces lots of disorders and distinctions and it drives down quality. 05:30.400 --> 05:31.800 But in our case, it was different. 05:31.800 --> 05:33.560 This was not a gate. 05:33.560 --> 05:40.680 So the feature growth, commit by commit on mainline, but reviews were happening on the whole 05:40.680 --> 05:43.800 feature on mainline. 05:43.800 --> 05:49.160 Now to avoid any context switching for the team, while we decided that whenever someone 05:49.160 --> 05:56.200 finished work before starting new work or at the start of the day or at the start of the afternoon, 05:56.200 --> 06:02.320 well, they would first check the to review column to see if nothing was waiting to be reviewed. 06:02.320 --> 06:06.760 And if something was waiting to be reviewed, well, they would review it. 06:06.760 --> 06:09.760 There was only a single rule in place. 06:09.800 --> 06:17.320 That was, no one was allowed to review a feature they were involved in writing. 06:17.320 --> 06:20.920 It had always to be reviewed to buy another engineer. 06:20.920 --> 06:26.240 There were no rules in place regarding who could review the code of who. 06:26.240 --> 06:28.320 There was no hierarchy in place. 06:28.320 --> 06:30.640 There was true peer reviews. 06:30.640 --> 06:37.960 And so we had seniors reviewing the code of seniors and reviewing the code of juniors as one would expect. 06:37.960 --> 06:44.840 But we also had juniors reviewing the code of other juniors and of seniors. 06:44.840 --> 06:50.280 Now I wasn't expecting that those juniors would provide many commands, but I was hoping 06:50.280 --> 06:56.000 that by reading the code of more experience engineers they would learn something. 06:56.000 --> 06:59.800 Although I have seen quite some interesting reviews coming from juniors because they asked 06:59.800 --> 07:04.880 different questions, they asked more outside the questions, the wide questions. 07:04.880 --> 07:08.960 So the necessary triggers for improvement. 07:08.960 --> 07:14.520 Well, if we don't practice pool requests or we don't practice bare programming, well we 07:14.520 --> 07:20.360 end up in a situation where unreviewed code is going straight into the removed mainline. 07:20.360 --> 07:27.760 And so we end up with having reviewed code and unreviewed code setting alongside each other on 07:27.760 --> 07:28.760 mainline. 07:28.760 --> 07:34.200 And so we have this piece of code is releaseable and this piece of code is not releaseable at the same time. 07:34.200 --> 07:42.760 How do we keep them apart and what this releaseable even mean in that situation? 07:42.760 --> 07:46.920 And that is the second key idea behind in unblocking code reviews. 07:46.920 --> 07:51.120 Stop the code review to be definitive for the release. 07:51.120 --> 07:57.240 So the code review doesn't define if a code base is releaseable anymore. 07:57.240 --> 08:03.080 It is not a gate anymore, it is not a go no go decision anymore. 08:03.080 --> 08:10.840 So the team practiced true continuous integration with string-based development that later 08:10.840 --> 08:16.160 evolved towards continuous delivery with a deployment pipeline. 08:16.160 --> 08:21.920 And every commit that went successfully through the whole deployment pipeline, through 08:21.920 --> 08:28.160 every stage of the deployment pipeline that have been totally tested, had the potential 08:28.160 --> 08:31.520 to arrive into production. 08:31.520 --> 08:37.040 So the gunband board was working on the level of the feature and reviews were working on 08:37.040 --> 08:42.680 the level of the features, but the software grow small commits by small commit on mainline 08:42.680 --> 08:48.760 and every commit could end in production. 08:48.760 --> 08:55.200 So this means that unreviewed code could end up in production whether the feature was finished 08:55.200 --> 08:59.560 or not, whether the feature was reviewed or not. 08:59.560 --> 09:05.120 So we can argue, yeah, but we are running the risk of having bad code quality in production. 09:05.120 --> 09:11.000 And yes, that is true and that happened many, many times. 09:11.000 --> 09:13.320 And that is not a problem. 09:13.320 --> 09:17.640 Because that quality is not a bug. 09:17.640 --> 09:19.960 Good reviews are not there to catch bugs. 09:19.960 --> 09:24.760 Anyone who thinks that a code review will catch a bug, well, it believes in fair details. 09:24.760 --> 09:28.360 It doesn't work like that. 09:28.360 --> 09:32.800 For catching bugs, we have automated tests and we have exploratory testing. 09:32.800 --> 09:38.320 So we had team commitment that any change had to be covered by an automated test, prefer 09:38.320 --> 09:39.800 a unit test. 09:39.800 --> 09:46.120 On top of that, we had a decent set of what we call automated acceptance tests, what people 09:46.120 --> 09:52.360 usually call into and test, although in our case, we are not end-to-end anyway. 09:52.360 --> 09:56.880 Then we also had the system in place that I would call peer testing, so we didn't 09:56.880 --> 10:00.240 have test engineers in the team, unfortunately. 10:00.240 --> 10:04.760 So the best thing we came up was that any feature that was implemented had to be tested 10:04.760 --> 10:10.360 by another engineer, not involved in implementing the feature, like performing exploratory 10:10.360 --> 10:11.360 testing. 10:11.360 --> 10:17.320 On top of that, we had a decent set of static code analyzes that would break the 10:17.320 --> 10:20.560 build on any rule violation. 10:20.560 --> 10:27.080 So the risk of working like that is not that, well, the code will be bigger or not, 10:27.080 --> 10:31.800 but that we might deliver lower quality code into production. 10:31.800 --> 10:40.240 But low quality code does not impact product delivery and even less functionality. 10:40.240 --> 10:47.080 On the other hand, low quality code does impact the speed of change, so it will impact how quickly 10:47.080 --> 10:52.480 we can implement new functionality and introduce new changes, and so it will impact our 10:52.480 --> 10:58.520 time to market, and that is the reason why we want to have high code quality. 10:58.520 --> 11:04.200 Does this not work? 11:04.200 --> 11:11.720 So any end-to-siscoet that works, because the test says so, it passed all the tests, but that 11:11.720 --> 11:13.000 could be better. 11:13.440 --> 11:16.440 But that is somehow always true. 11:16.440 --> 11:23.480 So code reviews will only reduce the chances of low code quality, but it will not, and 11:23.480 --> 11:28.720 that will mostly depend on the quality of the code review, but it will not eliminate 11:28.720 --> 11:30.640 that code quality altogether. 11:30.640 --> 11:36.360 Now, this worked for the steam, because while the steam decided that every single 11:36.360 --> 11:43.200 commit had to be reviewed at some point in time, and whenever an issue was raised, it 11:43.200 --> 11:49.640 had to be handled with the highest priority, like we dropped work, to fix the issue so that 11:49.640 --> 11:52.400 the issue was removed as quickly as possible. 11:52.400 --> 11:54.880 Now, the benefits. 11:54.880 --> 11:59.120 Now, in order to understand the benefits, we need to understand the concept of transaction 11:59.120 --> 12:00.760 cost, from lead manufacturing. 12:00.760 --> 12:06.320 So transaction cost is the cost of moving a batch of work from one stage to another stage. 12:06.320 --> 12:10.400 It's important to understand this, because while the higher the transaction cost is, 12:10.400 --> 12:13.960 the more inventories created in front of the next stage. 12:13.960 --> 12:20.040 So let's say, we want to order some goods online, and the transaction cost, so the cost 12:20.040 --> 12:22.440 to deliver the goods is three euros. 12:22.440 --> 12:26.040 Well, in that case, we will not order a single good of one euro. 12:26.040 --> 12:30.880 Now, we are going to order a new people of goods, so that we end up with a total cost, 12:30.880 --> 12:36.480 that is, a new people of the transaction cost, to compensate the transaction cost. 12:36.480 --> 12:45.160 So if it takes 10 minutes to create a change and a change is waiting for a code review, and 12:45.160 --> 12:54.120 it takes 2 hours to receive a code review, it means that the change was waiting for 92% of 12:54.120 --> 12:56.640 the total lead time. 12:56.640 --> 13:02.360 Now, this is a pretty expensive process, isn't it? 13:02.360 --> 13:12.700 So it signals us that good reviews are costly, and so in order to compensate the transaction 13:12.700 --> 13:19.360 cost of the code review, well, we are incentivized to ask less often for good reviews. 13:19.360 --> 13:24.120 And so what happens when we ask less often for good reviews, well, we are going to create 13:24.120 --> 13:29.120 more changes, people asking for the good review. 13:29.120 --> 13:36.880 And so, it discourages us to work in small incremental steps, and it discourages us to perform 13:36.880 --> 13:37.880 refactoring. 13:37.880 --> 13:41.440 And if we don't refact, then we are not paying back to any code app, and if we are not paying 13:41.440 --> 13:45.320 back to any code app, then this is going to end back to speed of change, and so introducing 13:45.320 --> 13:48.680 new functionality will take longer. 13:49.680 --> 13:55.480 Now, because we are waiting for someone to be available in order to do the code review, 13:55.480 --> 14:00.360 we are also incentivized to create more work in progress, because while people want to 14:00.360 --> 14:06.280 feel productive, they are not just going to wait there, until someone is available for 14:06.280 --> 14:08.280 a good review and do noting. 14:08.280 --> 14:11.280 So what will they do? 14:11.280 --> 14:14.800 Right, they will start new work. 14:14.800 --> 14:19.960 And there we have the start of contact stretching and multitasking, and we know where 14:19.960 --> 14:22.680 this goes, and it's not pretty. 14:22.680 --> 14:27.680 Now, because of little slow, the more work in progress we have, while the longer the 14:27.680 --> 14:33.760 lead times become, and so the more time it takes to get something onto the market. 14:33.760 --> 14:37.360 And so we end up with delay, delay, delay feedback. 14:37.360 --> 14:42.640 Learning will desalert, we become risk-averse innovation falls flat, and we go back to 14:42.640 --> 14:47.840 conserve the solutions, and this is what happens with organizations that practice pull 14:47.840 --> 14:48.840 requests. 14:48.840 --> 14:53.280 They go back to conservative ways of working. 14:53.280 --> 14:58.440 So to get things out sooner, while we need to reduce the transaction cost, and so we need 14:58.440 --> 15:02.360 to reduce the cost of the code review. 15:02.360 --> 15:07.240 So the benefits, second time. 15:07.240 --> 15:11.480 So in our case, the transaction cost was nearly zero. 15:11.480 --> 15:16.200 It had no impact at all on the total lead time. 15:16.200 --> 15:22.760 Commets already went through the deployment pipeline, and when it was successful, it could 15:22.760 --> 15:28.240 already be in production even before a code review happened. 15:28.240 --> 15:36.120 And so there was no reason for us to create more changes before asking for a code review. 15:36.120 --> 15:41.560 So we could work in small incremental steps, and refactoring happened naturally, and so 15:41.560 --> 15:45.960 we ended up with better code quality. 15:45.960 --> 15:52.680 Now the fact that un-reviewed code could already be tested, and could already be in production 15:52.680 --> 15:57.760 before a code review even happened, was surprisingly a significant benefit. 15:57.760 --> 16:02.560 Because now we already received feedback from testing, and we already received feedback from 16:02.560 --> 16:09.560 the production from how they used the feature or does not use the feature before we started 16:09.560 --> 16:10.560 the code review. 16:10.560 --> 16:14.400 Eventually, we didn't even have to start the code review if the feature was not used in 16:14.400 --> 16:15.400 production. 16:15.400 --> 16:20.400 We should have realligated the feature. 16:20.400 --> 16:25.120 Now because we never have to wait for someone to be available, we can work on functionality 16:25.120 --> 16:30.120 and one go from start to end, so we can focus on one single thing. 16:30.120 --> 16:37.680 We never start new work before finishing an ongoing work, and so we have no multitasking anymore. 16:37.680 --> 16:42.520 And therefore we also have way less work in progress, and because of little slot, well 16:42.520 --> 16:46.840 we have a reduced lead time, we reduced time to market. 16:46.840 --> 16:52.360 Now sometimes it happened that during a code review, while we discovered that this design 16:52.360 --> 16:57.200 was completely off, like completely wrong. 16:57.200 --> 17:01.600 This were the times where I was like on which we did some pair programming to avoid 17:01.600 --> 17:02.600 the situation. 17:02.600 --> 17:08.640 But then again, it wasn't a big deal because while the test set functionality worked, 17:08.640 --> 17:13.840 it was already in production, delivering value to our users, the users were already using 17:13.840 --> 17:14.840 it. 17:14.840 --> 17:21.840 So we had all the time to redo the design and do a total of rewrite, and we could even use 17:21.840 --> 17:26.600 a feedback from production to incorporate it in the rewrite. 17:26.600 --> 17:30.000 I think that is very difficult with pull requests, with pull requests we have two options 17:30.000 --> 17:31.600 when the design is off. 17:31.600 --> 17:37.280 Either we impose a fix, and during the time we fix it, while the feature is still blocked. 17:37.280 --> 17:41.600 We still have no feedback on whether the thing that we just implemented is the writing, 17:41.600 --> 17:44.600 we could even be wrong, but we don't know it. 17:44.600 --> 17:51.360 Or, well, there is delivery pressure, and so we cut corners, and we let it go, and we say 17:51.360 --> 17:56.320 we'll fix it later, and later, never happens. 17:56.320 --> 18:02.160 So in other case, no pressure, no stress, no fatigue, no burnout. 18:02.160 --> 18:08.560 Now because, and review the commits, goes straight into the remote mainline, and eventually 18:08.560 --> 18:13.840 arrive into production, it puts a lot of trust on the team. 18:13.840 --> 18:19.000 So we are expecting the team to do the writing instead of thinking that the team is stupid, 18:19.000 --> 18:20.840 and we need a process to fix for that. 18:20.840 --> 18:26.120 No, we were expecting the team to write automated tests, that they were going to write 18:26.120 --> 18:30.960 the write automated test, and that they would write high quality automated test, and 18:30.960 --> 18:34.320 an off automated test. 18:34.320 --> 18:40.440 So this is applying management theory, y over management theory x. 18:40.440 --> 18:46.680 And then again, well, if something wrong happens in production, that's also not the end 18:46.720 --> 18:51.920 of the world, as long as we have a process that allows us to fix it quickly, and in our 18:51.920 --> 18:56.320 case, we could release it in less than an hour. 18:56.320 --> 19:03.560 Now, to be honest, well, there is a caveat with non-blocking code reviews. 19:03.560 --> 19:10.080 It might be difficult to introduce and regulate it industry, especially healthcare, where 19:10.080 --> 19:15.440 they require a gate in the form of a code review before going to production. 19:15.440 --> 19:20.200 Although, I can imagine ways of introducing it, but I never worked in healthcare, so 19:20.200 --> 19:24.560 I never tested the idea, so I won't share it right now. 19:24.560 --> 19:29.080 On the other hand, there are better ways to achieve the same, and this is bare programming 19:29.080 --> 19:31.000 and team programming. 19:31.000 --> 19:38.280 So with bare programming, the bare can sign over the commit, and this is the review at 19:38.280 --> 19:44.320 a moment of committing, and they can tag the change with the identifying of the engineers 19:44.320 --> 19:51.120 involved in the bare, and this is your evidence that a code review happened. 19:51.120 --> 19:59.920 Now, I still think that bare programming and team programming are a superior delivery option, 19:59.920 --> 20:04.760 but in many organizations, this is a cultural stretch to introduce. 20:04.760 --> 20:06.000 It's just difficult. 20:06.000 --> 20:12.000 It's people are not open to the idea, the whole social aspect that is involved. 20:12.000 --> 20:18.840 So in those cases, non-blocking reviews, strategies are significantly better than any 20:18.840 --> 20:25.160 of the alternatives, because we have no gates in place, because we are no gates, we have 20:25.160 --> 20:35.320 no delays, so deliveries are never blocked, we have shorter time to market. 20:35.320 --> 20:40.120 Because we have no waiting times, while people are not incentivized to start new work 20:40.120 --> 20:44.840 before finishing ongoing work, and so we have way less work and progress, and because 20:44.840 --> 20:51.920 of little slow, our lead times are reduced, and our times to market are reduced. 20:51.920 --> 20:56.400 Because we are not starting new work before finishing ongoing work, while we have less 20:56.400 --> 21:02.680 context switching, so we have no productivity curves, no multitasking anymore. 21:02.680 --> 21:11.400 We have way less stress and fatigue and burn and out. 21:11.400 --> 21:17.000 Also when we find issues and good reviews, while we have all the time to fix them, there 21:17.000 --> 21:22.360 is no pressure to fix, because there is no delivery pressure, it's already delivered. 21:22.360 --> 21:29.120 It's already in production, delivering value, and lastly, well, we are not incentivized 21:29.120 --> 21:35.520 to work in bigger batches, so we can work in small incremental steps, and refactoring 21:35.520 --> 21:41.360 happens naturally, so leading to way better code quality, which allows us to improve the 21:41.360 --> 21:47.560 speed of change, and again, reduce lead time and time to market. 21:47.560 --> 21:50.560 Thank you. 21:50.560 --> 22:06.680 How do you see from this one, we have a agent, a agent, creating all the code? 22:06.680 --> 22:08.680 Can you repeat it? 22:08.680 --> 22:10.680 Can you repeat it? 22:10.680 --> 22:12.680 Can you repeat it? 22:12.680 --> 22:13.680 Can you repeat it? 22:13.680 --> 22:15.680 Can you repeat it? 22:15.680 --> 22:16.680 Can you repeat it? 22:16.920 --> 22:21.000 Yeah, well, I still think you need to review the code, because you need to understand what 22:21.000 --> 22:24.600 the AI agent generated. 22:24.600 --> 22:25.600 I have no letter that. 22:25.600 --> 22:33.600 Well, people, yeah, otherwise you have an asset where you don't understand what happened, 22:33.600 --> 22:35.040 and this is a huge risk. 22:35.580 --> 22:42.480 Thank you very much. 22:42.480 --> 22:52.480 I'm going to tell you a nice question. 22:52.480 --> 22:57.480 And of course, it's late to not even go to the list, but not go to the list. 22:57.480 --> 23:01.480 Not to mention, I think, from some terms of standards. 23:01.480 --> 23:06.480 Yeah, so, good point. 23:06.480 --> 23:13.480 So I mentioned, it requires trust and the team. 23:13.480 --> 23:20.480 And if I understood the question right, you are asking what all the things are needed for this to work. 23:20.480 --> 23:25.480 And can this work in any team? 23:25.480 --> 23:27.480 Now, we were lucky with this team. 23:27.480 --> 23:29.480 They were really demanding. 23:29.480 --> 23:35.480 And they saw us like, oh, they are going to tell us what's the right thing to do. 23:35.480 --> 23:36.480 So this is helpful. 23:36.480 --> 23:42.480 It's a bit more difficult when you have, like, in very senior environments, 23:42.480 --> 23:45.480 that always work with pull requests to just introduce that. 23:45.480 --> 23:50.480 Because a lot is in the minds of, well, first we have always worked like that. 23:50.480 --> 23:57.480 And second, you have a trust that's all good, but in the end. 23:57.480 --> 24:02.480 I think it's a long work process. 24:02.480 --> 24:06.480 It's a lot of massage, I think. 24:06.480 --> 24:09.480 Thank you. 24:09.480 --> 24:11.480 Very quick. 24:12.480 --> 24:13.480 That's good. 24:13.480 --> 24:18.480 You make a change, but it turns out that needs to make quite a lot of changes forever code. 24:18.480 --> 24:21.480 So your commit isn't really very small when it starts becoming big. 24:21.480 --> 24:25.480 And then, most of the tests pass, but you also have to support 10 different platforms. 24:25.480 --> 24:27.480 So it's a part of where the configuration is also support libraries. 24:27.480 --> 24:30.480 And not all the tests pass along with that formula. 24:30.480 --> 24:38.480 Is there any way you can still move forward if, if you come guarantee that you can fix every platform. 24:38.480 --> 24:40.480 For my free support thing all the time. 24:40.480 --> 24:43.480 Maybe it can still use this workflow. 24:43.480 --> 24:45.480 Oh, yes. So yeah. 24:45.480 --> 24:49.480 So repeat the question. 24:49.480 --> 24:53.480 It's allowed for release on these platforms, but not on these platforms. 24:53.480 --> 24:56.480 Maybe because the testing says that. 24:56.480 --> 25:00.480 So we have fixed everything right now, because I want you to forward on something else. 25:00.480 --> 25:03.480 Which still makes treating them. 25:03.480 --> 25:04.480 So yeah. 25:04.480 --> 25:05.480 So what? 25:06.480 --> 25:10.480 So what if we don't have a single commit that no people commit. 25:10.480 --> 25:14.480 And we need to do loads of changes everywhere in the code base. 25:14.480 --> 25:18.480 But we are deploying in many platforms on many different platforms. 25:18.480 --> 25:19.480 Target platforms. 25:19.480 --> 25:21.480 So I would suspect. 25:21.480 --> 25:23.480 Yeah, I don't know, actually. 25:23.480 --> 25:25.480 Well, the thing is what? 25:25.480 --> 25:30.480 So this requires a deployment pipeline that acts as a gate. 25:30.480 --> 25:34.480 So if tests are failing, it just don't go to production. 25:35.480 --> 25:41.480 And the team implements, stop the line. 25:41.480 --> 25:45.480 And that means they drop work and fix the deployment pipeline. 25:45.480 --> 25:48.480 Because, well, it means we are not delivering value. 25:48.480 --> 25:49.480 Yeah. 25:49.480 --> 25:50.480 Let's leave it to this. 25:50.480 --> 25:51.480 Thank you very much. 25:51.480 --> 25:52.480 Thank you.