WEBVTT 00:00.000 --> 00:10.920 Welcome to day. So this is the first time garage is presenting at Boston. Usually it's 00:10.920 --> 00:17.400 either a counter or a likes. This time it's me. I'm not part of the original developer group. 00:17.400 --> 00:23.660 I'm actually more like a contributor and part of the Duffler Association, which is a shifurding 00:23.660 --> 00:30.020 the garage project. So I can't answer deep chord questions, but I've been handling the 00:30.020 --> 00:36.540 community channel for quite a while for people who know me. And I think it's interesting 00:36.540 --> 00:42.940 for us to have a point a checkpoint today after five years of garage on the way we are 00:42.940 --> 00:49.340 and what you can do with it. So let's get going. The fur just a small word on it. We are 00:49.420 --> 00:55.420 a French association. So it's just a group of people. We try to promote digital sovereignty 00:55.420 --> 01:02.460 and privacy. And of course that goes so hard to word self-hosting as an alternative to 01:02.460 --> 01:09.580 big cloud providers. This means that we, because we do self-hosting, we need resiliency. And that 01:09.580 --> 01:17.660 means that we get good at time and relatively low supervision and the low manpower that goes 01:17.660 --> 01:25.660 with volunteer work. For anybody that needs to catch up, why is garage exactly? Why is it 01:25.660 --> 01:34.460 different? Garage is a self-hosted drop-in replacement for the AWS S3 protocol. I mean, 01:34.460 --> 01:40.780 AWS Object Storage General, but also there's a ton of product now that use that protocol. 01:41.740 --> 01:47.660 It is a bit different because it's mainly implemented resiliency through geographical 01:47.660 --> 01:53.500 resiliency. So the idea was to use not to put stuff in it at our center. We've ridden and power, 01:53.500 --> 02:00.460 ridden and connectivity, and instead to use smaller machines in a location that do not have 02:00.460 --> 02:07.340 redundancy, like re-appartment or like a shared location. And to implement the resiliency on 02:07.340 --> 02:13.820 top of that, so meaning that you have multiple geographically copy of it out of, and then if one 02:13.820 --> 02:19.100 location goes down, then you can just access it in another location. So you can use it to 02:19.100 --> 02:25.980 host a website. We host a defler, more than 600 websites on garage for quite a while now. 02:26.700 --> 02:32.700 You can store media, pictures, videos, as well. You can use it for peer cube as a backend. 02:33.660 --> 02:39.580 And a ton of people are using it as well as a backup target. So classical S3 protocol usage. 02:41.340 --> 02:45.980 Why is garage different? There's a ton of over-esprimplementation order. 02:46.860 --> 02:51.180 Garage is a bit different because it's coordination free. Meaning that we don't have a central 02:52.140 --> 02:58.060 elected master. There is no election going on. Every request, basically, we need to get consensus 02:58.140 --> 03:02.380 from the cluster. And that makes it extremely resilient to latency. I'm going to talk a bit about 03:02.380 --> 03:08.380 that later. But that's the main difference. We do not have a raft. We do not have elections. 03:08.940 --> 03:17.340 And that also creates some constraints on how the clusters are managed. But the benefit is that 03:17.340 --> 03:23.180 it's extremely beneficial for a situation where you have high latency between nodes. 03:23.740 --> 03:28.780 Because the lack of elections will mean that basically every request is going to give you the 03:28.780 --> 03:39.180 base latency. This is a quite old graph at this point. I didn't rerun the testing, but the point 03:39.180 --> 03:47.500 still stands. Basically, here what is shows you is that when you operate on objects in garage, 03:47.580 --> 03:54.140 when you're going to get latency, when you have high not-to-not-latency. We're talking about 03:54.140 --> 04:00.140 a hundred milliseconds. This is not that a central latency. This is like almost Europe and US latency. 04:01.180 --> 04:06.700 You get much better latency on some operations. And that's what really where 04:07.500 --> 04:12.860 architecture is about. Because other solutions are much more optimized if you want to just 04:13.340 --> 04:20.300 put stuff in Iraq and then have just like horizontal scaling that is very located physically speaking. 04:24.780 --> 04:32.860 So the consistency model is not acid. We only have read after write consistency, which is still 04:32.860 --> 04:39.340 better than eventual consistency. I think that's as well what's as free. Had initially another 04:40.300 --> 04:44.860 originally. I think they improved a bit for models since then, but they still say that they 04:44.860 --> 04:54.140 have we have to write consistency. It might be better. But yeah. We talked about geographical 04:54.140 --> 05:01.740 deployments. Garage has a zone concept that allows you to spread your deployments across different 05:01.740 --> 05:06.380 zones, which are geographical locations. And then you can have multiple nodes per location. 05:06.380 --> 05:11.020 So you can have, for example, free locations. And then have two nodes in one location, 05:11.020 --> 05:15.100 free nodes in another locations. I'll also do to scale a bit for cluster as well. 05:19.660 --> 05:23.820 What makes Garage a bit different? So I already talked about it. It's really the geographical 05:23.820 --> 05:32.140 distribution aspect. Here we can see an example that you could have with multiple zones. One in 05:32.220 --> 05:36.620 the UK, one in France, one in Belgium, one in Germany, and one in Switzerland. 05:38.780 --> 05:43.580 And you can see that the chunks of data are actually spread. So this cluster has a replication factor 05:43.580 --> 05:49.180 set to free. And you can see that the blue chunk is actually replicated in Belgium in France and 05:49.180 --> 05:54.380 in Switzerland. But the red chunk is actually replicated in Germany in France and in the UK. 05:55.180 --> 06:00.860 So on top of the cluster, you have a replication factor. And then that replication factor would 06:00.860 --> 06:04.300 basically govern all of the copies of that habit happens in the cluster itself. 06:08.460 --> 06:13.740 This is a quite old compatibility list. We have more now. But these are some of the software 06:13.740 --> 06:21.500 that you can find that are using S3 for storage. And that can be configured to use Garage. 06:21.500 --> 06:29.180 Again, because some software have hard-coated dependency to Amazon S3. They are not always 06:29.260 --> 06:35.020 compatible. But more and more software now are becoming compatible with self-hosted S3 alternative. 06:36.380 --> 06:41.500 So this is a very exciting. You can find more at today's list on the website. We have a section 06:42.140 --> 06:48.700 in here called the integrations where you can basically list all of the software has been tested. 06:48.700 --> 06:51.900 We've configured some tools for them to help you get started. 06:52.220 --> 06:59.820 So nothing was new at this point. All of this was from over Garage Talks. 07:01.100 --> 07:07.420 Now this is the new content. This is just a short history. We have a roadmap that we had so far. 07:07.420 --> 07:13.420 So the initial beta release was in 2021. We have the actual first release version that we 07:13.420 --> 07:20.540 talked about to use in 2022. And then we had the first table release in 2024. 07:20.540 --> 07:28.940 There was a nice talk done by Alex at first them. That was an onswing this. 07:30.860 --> 07:36.300 The one has been here for quite a while. We decided I was talking with Alex today. We decided to 07:36.300 --> 07:43.340 deprecate it this summer one year after we released the 2.0 version. Right now we're still maintaining it. 07:44.060 --> 07:50.620 And then the 2.0 was released a couple of months ago. It actually doesn't have a ton of 07:50.620 --> 07:57.580 breaking changes. But it has a brand new HTTP administration API, which is one of the reason why we 07:57.580 --> 08:06.140 had to release a new major version. And then we changed the old replication mode setting to split it 08:06.140 --> 08:12.060 between the replication factor and the consistency policy. So that made things a bit more configurable. 08:12.060 --> 08:17.660 It's for you to pick your replication and your consistency factor differently. 08:19.340 --> 08:25.580 We are thinking about releasing a free.0 version. We are really thinking about 08:25.580 --> 08:30.620 versioning support. Garage does not support right now as free versioning, which is a huge issue 08:31.420 --> 08:36.140 for a lot of software that relies on it. So there's a couple of things like that. 08:36.220 --> 08:42.780 Versioning, tags on buckets, object retention policies. Those are things we are thinking about. 08:42.780 --> 08:48.540 We are not making any promises, but they are or objectives for the freedom of the version. 08:51.580 --> 08:57.100 Okay, so one thing I wanted to do for quite a while is to spend some time on best practices 08:57.100 --> 09:00.860 for guard deployments. We have a ton of people asking your questions, so it's 09:01.580 --> 09:07.020 sometimes a bit clunky. So I have a couple of sites of that. And then we can 09:08.060 --> 09:14.460 either do a quick demo or I can take questions. So the things you should know, if you don't 09:14.460 --> 09:20.540 know garage yet, we don't have to support within Garage. So Garage allows you to configure 09:20.540 --> 09:27.580 a website in point. And that website in point is spinach dp. It is by design. We intend to 09:27.580 --> 09:32.620 do your own certificate management and outside of the garage. And so it means that you have to 09:32.620 --> 09:37.340 set up your own proxy. We have documentation for engineers, for caddy and a couple of hours. 09:38.620 --> 09:44.140 Usually it's not really a big issue, but it's something that we made that just consistently back 09:44.140 --> 09:50.060 back then. We don't have yet a free anonymous access, which goes for issue for some 09:50.060 --> 09:56.220 softwares. Basically, we advise you to use the website in point for that, but not also for 09:57.100 --> 10:02.380 match to use that way. So it's something that we're freaking about. I think there's an issue right 10:02.380 --> 10:07.500 now about it. And I was discussing with Alex, we might have a PR coming in couple of months. 10:09.180 --> 10:12.700 And it might require us to release any major version, we're not through yet. 10:12.700 --> 10:24.380 Due to the lack of, like I said, like cluster management or elections, we don't have a way 10:24.380 --> 10:30.060 to kind of like automatically configure the cluster. And so that rolls always fall back to the 10:30.060 --> 10:35.100 operator. So that means that when you create a cluster, the nodes can discover themselves, 10:35.500 --> 10:41.820 but they will not autoconfigure themselves, because the cluster layouts that you will configure, 10:41.900 --> 10:47.180 which determines which nodes does what, is something that needs to be done by some from a 10:47.180 --> 10:53.100 30, but we don't have. So this can be node. Sorry, this can be done by interacting with any nodes. 10:53.980 --> 11:01.420 But it falls back basically on an operator to do that. And that operator can be either human, 11:01.420 --> 11:07.580 you can do that manually with a CLI, or you can also build your own scripts or operator in the 11:07.580 --> 11:11.580 Kubernetes, so that's the way to do that for you if you want. A couple of people in the 11:11.580 --> 11:15.660 community have done that. And then you're going to make cluster set a bad way. 11:18.060 --> 11:23.180 Verification factor is something that is defined at cluster creation, and that we 11:23.180 --> 11:29.580 strongly advise not to change. They are in supported procedures to do that, but please don't try it. 11:30.460 --> 11:36.220 We had issues in the past, and it's very annoying to recover cluster in that state. So basically 11:36.300 --> 11:42.460 what we advise is really think about your replication factor first, and then if you need to change it, 11:42.460 --> 11:51.020 just slip in a cluster and copy your data. Come on, pitfall. The default region in garage is 11:51.020 --> 12:00.380 garage, not USD1. Very common issues. And then we had a discussion about access well yesterday, 12:00.380 --> 12:05.340 and it was saying that really from his own experience as well, salvaging clusters from people 12:05.420 --> 12:12.540 that may or else after disasters do not use the degraded consistency, but I see really only 12:12.540 --> 12:16.380 it for that I recovery. Don't use it for like a regular cluster. 12:18.780 --> 12:25.340 Well, I had a lot of questions along the last two years on what hardware people should use. 12:26.300 --> 12:33.500 We really bit garage to tell her any hardware, but it works best on some hardware, at least. 12:34.460 --> 12:37.420 Those things are pretty obvious. I think for most people in the storage community, 12:38.300 --> 12:43.180 but they might be not necessarily obvious for people that just want an S3 storage store, 12:43.180 --> 12:49.660 and they don't really know a lot about storage. So the first rule, when you start to your storage, 12:49.660 --> 12:56.060 is to not put your storage back in on something like an FS or an S&B. It's a reciprocal disaster. 12:56.060 --> 13:02.220 So please don't do that, especially from the Metadata folder. The base on S&B yarn and FS don't 13:02.300 --> 13:07.340 tend to perform really well, and sometimes they have bizarre issues, especially when you cut the 13:07.340 --> 13:16.540 network. So don't do that, please. Get a right-intensive flash disk for the Metadata folder. 13:16.540 --> 13:22.540 To be honest, like anything flash should work for like very, very small use case, 13:22.540 --> 13:26.700 but if you have any service use, and if you have a relatively big cluster, 13:26.700 --> 13:32.140 especially if you have nodes that come and go, garage actually use the Metadata database 13:32.860 --> 13:39.020 to store queues of blocks that needs to be received, and those queues can be very, very 13:39.020 --> 13:47.580 right-intensive. I can tell you that at the third, we had normal SSDs, like some some, like 13:47.580 --> 13:54.300 retail SSDs, and we wore them out in less than two years. They basically retot their, 13:54.940 --> 14:03.260 their wear leveling. So try to find something a bit flash, sorry, like geared towards a 14:03.260 --> 14:07.820 right-intensive workload, and keep an eye on the wear leveling of your drives. 14:09.980 --> 14:16.780 With vise too, as well use red one for that, and to use a copy on the right-fast system, 14:16.780 --> 14:22.460 to do stop plotting. You can also enable compression. You get a bit more latency, but it 14:22.460 --> 14:29.020 basically compressively well, most of the base does. Get HDDs of flash for your data folder, 14:29.580 --> 14:35.260 use XFS for your data folder, garage creates a lot of small files, so you need something that has 14:35.260 --> 14:43.420 a lot of eye nodes. And here you have a choice. We have a multi HDD mode for the best performance, 14:43.420 --> 14:49.180 so guys will basically take each drive independently, and then your application can be 14:49.260 --> 14:53.580 basically geographical, like you don't need to have multiple rebukers in the same zone, 14:54.540 --> 14:59.660 if you can retrieve them from another location. If you really don't want to think about it, 14:59.660 --> 15:05.340 you can use red, and you kind of red below it, ZFS, or and DDA more or the best, 15:06.700 --> 15:09.900 but you will leave a lot of performance on the table, because you're absolutely basically 15:09.900 --> 15:16.780 building a TV array. But on the CPU and the RAM, we don't need that much, of course, 15:16.860 --> 15:20.460 depends on the use case. If you have a cluster, we've one bit above data, 15:20.460 --> 15:28.700 having a bit of RAM will help for metadata. Above metadata, we have multiple metadata 15:28.700 --> 15:35.580 engine in garage. Basically, the metadata is free things that's all of the mapping between the 15:37.580 --> 15:43.180 files and the blocks. It's also the bucket, the bucket, the access on all of that stuff. 15:44.140 --> 15:48.300 But also, any file that you upload to the cluster, well, are below three kilobytes, 15:48.300 --> 15:53.420 will actually be stored in the metadata and engine itself, and not on disk. So it's something 15:53.420 --> 15:58.140 good to know if you store tons of small files, you might not see those blocks in the data 15:58.140 --> 16:04.780 folder, there would be a metadata side. We had multiple engine, along with yours, 16:04.780 --> 16:11.260 sled has been removed. After one dot zero, if you haven't done that, you need we need to migrate. 16:12.220 --> 16:17.260 We have the tools for that. It's basically take your no-down, execute the migration, and then 16:17.260 --> 16:23.580 put it back up. The two means that we have right now are SQLites, which is the safest one. 16:24.380 --> 16:29.340 It is recommended for smart clusters. It has a couple of performance issues, but it works, 16:29.340 --> 16:32.940 it's pretty rock solid in terms of power loss and that I have consistency. 16:33.740 --> 16:39.820 And then anundibi is a bit faster than SQLite. It's there for big clusters. However, we had a 16:39.820 --> 16:48.380 couple of plenty issues a long time ago, and if you have an unrollable power, then I strongly suggest 16:48.380 --> 16:56.220 you use a UPS with anundibi over to backup frequently. Anundibi has also a limit for the key 16:56.220 --> 17:02.540 size that has been hidden by a couple of people using backup services. So the maximum length 17:02.540 --> 17:07.740 of a usable key is four hundred and eighty bytes with anundibi. So if you need long case, 17:07.820 --> 17:15.020 you might have to use a SQLite. And then we had a newcomer, which is FJL, which is experimental 17:15.020 --> 17:20.300 of this point, but it's already in the code base, so you can try it and let us know what you think. 17:22.140 --> 17:27.340 The method that engine can be downloaded, like there is no really restrictions. You can have a 17:27.340 --> 17:37.180 couple of nodes using a SQLite and a couple of nodes using anundibi. We really thought 17:37.260 --> 17:44.060 garage to be something that was meant to be your distributed, but following the Minio people 17:44.060 --> 17:49.820 turned off hard. It's great like that. We had a ton of people asking us for single load deployments, 17:49.820 --> 17:56.460 which are a bit clunky, because it's really meant to be a cluster. So we, of course, if you have 17:56.460 --> 18:02.140 a single load, you will lose resiliency. Make sure you have backup for a metadata. If you use 18:02.140 --> 18:06.140 anundibi with single load and you have a poor cut, you might end up with a corrupted database, 18:06.780 --> 18:11.340 and it's very annoying to recover from that. We really don't have a way to rebuild the metadata 18:11.340 --> 18:16.460 from the data. So you will have to basically, you will potentially lose some data if you 18:16.460 --> 18:22.220 did that. Like you will at least lose a lot of the file names and any file that might be stored. 18:23.260 --> 18:28.860 So we advise you to use SQLite if you did that. And potentially put a UPS on top. Some people 18:28.940 --> 18:35.180 really have super flaky power, and they are not really crushed like every two days, and the chances 18:35.180 --> 18:41.820 for you to have a corruption with that is relatively high. Right now, the experience to set up 18:41.820 --> 18:45.980 garage in a single load is a bit long. There's a ton of components that have that needs to be 18:45.980 --> 18:52.380 done. Someone from Recommonity actually put together a bit of a wrapper for that. We might actually 18:52.380 --> 18:58.380 integrate that into garage itself. So I was discussing with Alex yesterday and we were thinking 18:58.380 --> 19:03.500 about integrating like a garage, serve, dash, dash, single load that would basically do a 19:03.500 --> 19:09.340 single load setup. So that might come into following the move. With the node deployments, 19:10.700 --> 19:18.140 Joe distributed zones tried to have at least free. Keeping mine your network and Io, we had people 19:18.140 --> 19:24.540 built a very, very big clusters on top of DSL effectively, and then they lose a node and they 19:24.620 --> 19:30.940 have to build it and it takes literally half a month. So be careful about that. You can easily 19:30.940 --> 19:36.700 accumulate data and then keep an eye on how long it would take for you to recover from that. 19:37.900 --> 19:43.020 Please set up monitoring if you don't have that, it's very hard to understand what's happening. 19:43.020 --> 19:48.780 You can get some inside, but it's difficult. To give you a bit of an ID, Duffer has been running 19:48.860 --> 19:53.980 a nine-terbat cluster, which is three nodes, three nodes and two nodes. So three locations, 19:54.620 --> 20:01.580 replication set to three. Over regular fiber across Europe, the between Lyon and Burstall, 20:01.580 --> 20:08.620 the bit of Britannia as well. We heard that there are tons of bigger clusters out here, 20:09.660 --> 20:14.460 up to the petapites, but we don't really have a lot of data because we don't have any telemetry 20:14.460 --> 20:19.100 in the project by design, so we don't know. Tell us if you have big clusters, we're interested. 20:21.900 --> 20:26.220 I'm going to go a bit faster, even. I don't think we're going to have time for a demo. 20:27.820 --> 20:34.140 But we, on deploying, really use whatever you want, garage is a single binary. We compile it, 20:34.140 --> 20:39.660 we went to Great Land to compile it statically, so you can really just drop it on the system. 20:40.620 --> 20:47.500 You can also use Docker. We have Kubernetes and console integrations for not discovery, 20:47.500 --> 20:53.500 but you will have to set them alive manually or try to script on an operator to do that. 20:55.020 --> 20:58.780 You can use something called Gateway nodes, which don't store anything, but can kind of 20:58.780 --> 21:05.980 like geared your request toward the best node for load balancing. And if you find the default settings 21:06.780 --> 21:11.900 or default missing time to be a bit conservative, you can adjust the missing tranquility, 21:11.900 --> 21:16.940 which is basically how much time a garage sleeps between missing operations 21:16.940 --> 21:21.500 to avoid overloading the cluster as a whole. There is a Kubernetes storage controller 21:21.500 --> 21:25.660 that is in the work as well from the community, so you can take a look at that if you run into 21:25.660 --> 21:33.500 Kubernetes. We have someone from the community that created a UI this year. It's already usable. 21:33.500 --> 21:39.420 It's delivered to the admin API. So thanks to them. I forgot to put the name in here. I should 21:39.420 --> 21:48.060 have actually. But it's not here. You can find it. And then we also have an official UI coming 21:48.060 --> 21:54.060 later this year, but it will be embedded within the product itself. So those are mockups for now. 21:54.060 --> 22:00.300 But this is a very actively being worked on things to some friends that you got from 22:00.300 --> 22:08.940 an internet this year. So Valam, I want to do short paths on the garage metrics. So this is what 22:08.940 --> 22:13.500 to get if you're a version of a default dashboard. Really, the two things that you need to be aware 22:13.500 --> 22:18.860 are those two things here. So the racing queue length and the racing overall blocks. It will 22:18.860 --> 22:24.620 still shut up. It means that you are issues because your clusters cannot sync blocks correctly 22:24.620 --> 22:32.220 across the clusters. The rest are not necessarily an important but they give you more 22:33.580 --> 22:39.580 insights on the information. One thing that a lot of people don't know is that garage for safety 22:39.580 --> 22:44.620 reasons don't delete blocks right away if you delete a file. So if you delete a file in this 22:44.620 --> 22:50.460 spray PI, a garage will actually wait on those 24 hours before actually removing the block from 22:50.460 --> 22:55.100 storage. So if you create and remove a lot of files, then potentially you might see your storage 22:55.100 --> 23:00.940 going up for a while because garage will not remove those yet. So the the table gcq length 23:00.940 --> 23:05.740 has a bit of that as well. Like it was, show you the blocks as very remove black garage across time. 23:08.380 --> 23:12.940 One thing is going wrong. Enable for debug logs. We had a ton of people with 23:12.940 --> 23:18.700 authentication issues because of the S3 way of signing a request. So be careful about your 23:18.700 --> 23:25.420 reverse proxy configuration. Check the tranquility if you think the racing's are slow. And then 23:26.620 --> 23:32.620 LMDB has a bad tendency to kind of like being blown it up a time goes on. You can just shut down your 23:32.620 --> 23:40.700 garage note, do an MDB copy with the C parameter to compact it and then it will go back to 23:41.020 --> 23:50.300 normal size. We take questions on the garage matrix channel in English and you can log issues as well 23:50.300 --> 23:57.100 on gith.daflow.fr, which is a 4G Joe instance. We also have a mirror of a code on github but it's 23:57.100 --> 24:02.300 basically just here for the people that want it on github and we don't take issues there. 24:03.340 --> 24:07.020 And then if you open an issue, if you have any problem, please provide the output of 24:07.020 --> 24:12.620 garage status and then garage status as well. If you're moving from main to main, I don't think 24:12.620 --> 24:16.140 I will have done to do a demo. But basically, the idea is that you need to list you 24:16.140 --> 24:21.740 to break it on keys and then you create both buckets on the garage cluster. Right now, you will 24:21.740 --> 24:26.300 have to create new keys because garage enforces that the keys start with the specific string. 24:27.820 --> 24:35.820 We will put a PR soon to basically add a parameter to skip that and use garage key import to 24:35.900 --> 24:40.540 basically import your own key and then you just loop over the buckets with our clone and then 24:40.540 --> 24:45.580 you just copy for that up. I thought blog posts would actually be better for this as I don't have 24:45.580 --> 24:53.180 time for a demo here. And that's it, I think. I can do some demo. 24:58.700 --> 25:01.260 Does that include questions? Does that include questions? 25:01.260 --> 25:04.380 Yeah, that's your whole smart. Okay, so that's five minute for questions. 25:14.220 --> 25:20.380 Yeah, so the question is do you need gateway nodes within a cluster? The response is no. Any nodes 25:20.380 --> 25:28.140 that can take request. Gateway nodes are only here for helping. Basically, they have a copy of 25:28.300 --> 25:34.780 the metadata and the cluster and they are able to direct. So what might happen if you hit any nodes 25:34.780 --> 25:38.620 is that you will bounce like the node might not have a copy of it at all and it will might go elsewhere. 25:38.620 --> 25:43.980 And so you lose one RTT in latency. What you can do is you can run a gateway locally. 25:43.980 --> 25:47.740 And then that way you can just talk to the gateway locally and then the gateway will basically 25:47.740 --> 25:52.220 ban you directly to the correct nodes. So that's much faster in that regard. 25:53.180 --> 25:58.700 One caveat though is that gateway nodes are fully part of the cluster itself. 25:58.700 --> 26:02.460 And so they basically have full access to the entire cluster. So that means that 26:02.460 --> 26:07.180 whatever if you're running a gateway locally, then you have to trust that that compute 26:07.180 --> 26:12.220 to because it basically has full embedded access to the entire cluster. Yes? 26:12.220 --> 26:19.420 And you may come that it's more than a consumer. I'm guessing if it's doing a single one 26:19.420 --> 26:25.020 price, do you have any info on what's the block size for those price? So you can optimize 26:25.020 --> 26:33.820 that best. Okay. So the question is, we're doing a lot of block rights. And synchronous 26:33.820 --> 26:38.460 we're back rights here. How to optimize that? Especially if you have something like ZFS but 26:38.460 --> 26:45.340 has a block size. So that's a good question. I think I don't know what LNDB used by default. 26:45.340 --> 26:50.540 I would assume it's something that's 16K or similar. That's usually common for databases. 26:51.180 --> 26:56.620 For the metadata, so that depends a bit on what you pick. For the data itself. 26:56.620 --> 27:05.580 So garage has a block size setting. And it will effectively split any incoming file up to that 27:05.580 --> 27:09.660 block size. So if you have a block size, it says to set to I think the default is one megabyte. 27:10.700 --> 27:15.660 Any file that is below one megabyte will start in the single chunk. So it will appear as one file 27:15.660 --> 27:22.060 on your data folder. And if you have a file that is above that, like then it will be split in 27:22.060 --> 27:27.580 multiple chunks. We actually advise to raise that block size because there's no rental 27:27.580 --> 27:32.060 consequences. The only consequences are the additional buffers because when garage 27:32.700 --> 27:37.980 under the block, it's 101 by one. So the bigger you block, the more memory you need. And also 27:37.980 --> 27:45.340 like within RP cycles because it's brought by block, then your RP cycles will last longer. But that's 27:45.340 --> 27:53.180 more or less it. So big clusters really go up in in block size between sometimes like 10 megabytes 27:53.180 --> 27:57.980 or even 15 megabytes. If you store a lot of big content, like people storing video content, 27:58.540 --> 28:02.060 it can go very high. And that gives in a lot of more bandwidth for that. 28:08.060 --> 28:13.340 So the question is what is the metadata folder used for? And it's really only used for that. 28:13.340 --> 28:17.180 And a couple of static configuration for like the layout configuration as well. 28:20.780 --> 28:24.780 So the metadata folder needs to be basically tuned for the metadata engine that you use. 28:28.380 --> 28:31.980 You mentioned that what's currently directly deleted, where a file is deleted, 28:31.980 --> 28:35.020 is possible to validate the file will be updated. 28:36.780 --> 28:42.700 So the question is blocks are not necessarily deleted right away when you delete the file in 28:42.700 --> 28:47.100 this free. Is it possible to recover that? Right now it don't think it is. I mean technically 28:47.100 --> 28:52.460 the blocks are still on disk, but the metadata has been deleted. So I mean one thing though is 28:52.460 --> 28:57.260 if you have a metadata task snapshot, then you would be able to restore that and then to just 28:57.340 --> 29:00.620 give a block to garage and then we scan for blocks and it will just connect again. 29:07.500 --> 29:12.780 So the question is did we consider implementing a recycle bin? This basically would be implemented by 29:12.780 --> 29:14.700 SV versioning. Yes? 29:17.100 --> 29:22.860 If you are implementing versioning, we also implement object or 29:22.860 --> 29:32.220 block. So the question is if we implement versioning, we will we be also implementing 29:32.220 --> 29:37.180 implementing object locking. I need to look into it. I think it's something that we haven't done yet, 29:37.180 --> 29:43.740 but that's something that's required, why not require it, but is high in demand by a lot of backups 29:43.740 --> 29:50.220 software that is it for a ransomware protection. So yeah. And the times up, you can find me, 29:50.220 --> 29:56.860 I will be hanging around. You can find me a couple of questions.