WEBVTT 00:00.000 --> 00:20.000 We are going to talk about how we did playing, how he did playing the collaborative 00:20.000 --> 00:22.000 team, his wages and production. 00:22.000 --> 00:36.000 All right, folks, there was a lot of delay, sorry about that. 00:36.000 --> 00:44.000 So, let's start. How many of you have used YJS or any other CRDD algorithm in production? 00:44.000 --> 00:52.000 And how was your experience like did you face any issues at the start or was it all smooth sailing since 00:52.000 --> 00:59.000 the start? So, we are going to talk about that, how we did it at playing and let me start 00:59.000 --> 01:07.000 with a short story. So, last year we upgraded one of our core dependencies from a major version of 01:07.000 --> 01:15.000 grid. And guess what? The next day everything was on fire, all the servers were crashing. 01:15.000 --> 01:22.000 The Kubernetes was just restarting the pods again and again and traffic was redirecting. 01:22.000 --> 01:29.000 It was chaos, but the best thing about CRDDDs was that our users didn't face any of it, 01:29.000 --> 01:34.000 like they didn't lose a single bit of data and it was all super awesome. 01:34.000 --> 01:41.000 So, that's what we are going to talk about the past, the present and the future. 01:41.000 --> 01:52.000 So, yeah. So, YJS? Well, it's magic, but magic means more fees long. 01:52.000 --> 01:57.000 So, yeah, a bit more about playing. 01:57.000 --> 02:05.000 Playing is an open source project management tool. We have projects, wiki and air, everything in one workspace. 02:05.000 --> 02:09.000 And you can sell for a set, air gap it, run it wherever you want. 02:09.000 --> 02:23.000 And wiki is where the YJS lives. So, a lot of people get this a bit wrong when they start with any kind of CRDD. 02:23.000 --> 02:29.000 Especially I've seen this in a lot of forums where people are super confused about this. 02:29.000 --> 02:33.000 So, I wanted to start with this. 02:33.000 --> 02:47.000 So, what's your idea of storing the YJS stuff like? Do you want to store the JSON or also the metadata? 02:47.000 --> 02:53.000 What do you think? To get the collaborative part in action. 02:53.000 --> 02:59.000 A lot of people think that we can just get away with storing JSON and HTML on the server. 02:59.000 --> 03:04.000 And they expect that collaboration will work, but that's not the case. 03:04.000 --> 03:09.000 And while I implemented it, luckily LLM passed through the docs well. 03:09.000 --> 03:16.000 And I got to know that here I have to store the binary and not just the JSON. 03:16.000 --> 03:24.000 So, yeah, we like wideboarding a lot at play and the aesthetic. 03:24.000 --> 03:35.000 So, previously when we started, we only had a normal restrict set up in which you could not really collaborate. 03:35.000 --> 03:37.000 And for that, we were using JSON. 03:37.000 --> 03:47.000 But, with time as we wanted real-time collaboration and we did an on-demand migration to wideoc as users started loading the pages. 03:47.000 --> 03:54.000 And let me give you a nice culture on this. 03:54.000 --> 03:58.000 So, when you do this for the first time. 03:58.000 --> 04:04.000 So, as you can see over here, the server needs to be converted into binary. 04:04.000 --> 04:09.000 And if you want the local first experience, you use index TB as well. 04:09.000 --> 04:20.000 So, as the client converts the JSON into binary and stores it into the index TB of the client, everything is all good. 04:20.000 --> 04:31.000 But say, if the client doesn't really make any edits, then the underlying library that we use, that's focus focus. 04:31.000 --> 04:33.000 That's the wisest web socket back end. 04:33.000 --> 04:41.000 If the user doesn't make any change on to the document, it really doesn't trigger a save call of the binary. 04:41.000 --> 04:45.000 So, you can see how where we are going. 04:45.000 --> 04:54.000 So, if the user just leaves the page, comes back to it after some time, reloads it or anything. 04:54.000 --> 05:04.000 The server again in the server's logic, we have the check that if the binary is not in the database, we convert JSON into binary. 05:04.000 --> 05:15.000 So, but the client has the binary in their index TB, which was generated by the first visit. 05:15.000 --> 05:24.000 So, now on the client's new, on the second visit, the new binary and the old binary come into index TB. 05:24.000 --> 05:29.000 And wisest tries its best to merge things. 05:29.000 --> 05:38.000 And what we faced was someone having like 1000 paragraphs and it getting duplicated in a really weird way. 05:38.000 --> 05:50.000 So, the gacha is that you should store the binary immediately after conversion of the JSON for the first time, else there would be chaos. 05:50.000 --> 05:51.000 All right. 05:51.000 --> 05:55.000 So, the binary needs to be used as the source of truth. 05:55.000 --> 06:07.000 For all the other needs, say for crawling, for indexing and all that, you can use the derived views of JSON HTML, etc. 06:07.000 --> 06:11.000 And let me just, yeah. 06:11.000 --> 06:13.000 So, the migration code is very simple. 06:13.000 --> 06:22.000 You just have to encode the binary and store that in your database and load from here the next time. 06:22.000 --> 06:23.000 And all right. 06:23.000 --> 06:27.000 So, remember that fun little story that I told you about. 06:27.000 --> 06:31.000 That's when the server's catching firepot. 06:31.000 --> 06:34.000 So, let's dig into that. 06:34.000 --> 06:35.000 All right. 06:35.000 --> 06:42.000 So, it was a very nice Friday and focus focus V3 drops. 06:42.000 --> 06:46.000 We were super excited that multiplexing works finally. 06:46.000 --> 06:51.000 And there were a lot of issues in V2 that were fixed in V3. 06:51.000 --> 06:53.000 So, we were very excited. 06:53.000 --> 07:00.000 We did our homework, test data lot of stuff, did staging, QA, did everything. 07:01.000 --> 07:07.000 It was just two lines of change on our side of code when we ran the code mod. 07:07.000 --> 07:10.000 And we deployed it on a Friday. 07:10.000 --> 07:12.000 I mean, what could go wrong, right. 07:12.000 --> 07:18.000 So, we actually treated it as a minor release, but yeah. 07:18.000 --> 07:24.000 So, I even made an arrogant post on Twitter that, hey, software engineering is so. 07:24.000 --> 07:27.000 What you see is what you get types. 07:27.000 --> 07:32.000 And that weekend was chaotic. 07:32.000 --> 07:43.000 So, let me tell you what happened on that exact day and the couple of days following that. 07:43.000 --> 07:48.000 In just a couple of hours, our servers started crashing out of nowhere. 07:48.000 --> 07:52.000 And a huge customer also on boards on the same exact date. 07:52.000 --> 08:00.000 So, they are using massive tables, hundreds of rows and hundreds of columns and multiple such pages. 08:00.000 --> 08:05.000 And our memory just, our server just starts crashing like anything. 08:05.000 --> 08:10.000 And it goes into a huge loop of crashes. 08:10.000 --> 08:14.000 Well, you know, Kubernetes, it was just doing its thing. 08:14.000 --> 08:19.000 So, luckily, we had our fallback logic. 08:19.000 --> 08:26.000 So, this is super important that on the client side after if the server disconnects, 08:26.000 --> 08:28.000 then we do something like this. 08:28.000 --> 08:33.000 If it is disconnected, we fetch the binary from the database and then apply it on the client. 08:33.000 --> 08:40.000 So, in certain intervals, we were basically sinking the client's slowly. 08:40.000 --> 08:42.000 So, yeah. 08:42.000 --> 08:47.000 So, it basically converges and we were using this by the way. 08:47.000 --> 08:51.000 This was for a couple of months before we started using Hocus Progress. 08:51.000 --> 08:55.000 This was our primary way of sinking stuff. 08:55.000 --> 08:58.000 And it worked flawlessly thanks to VHS. 08:58.000 --> 08:59.000 And yeah. 08:59.000 --> 09:02.000 So, there was no data loss at all. 09:02.000 --> 09:06.000 So, suppose that incident, we started monitoring all these things. 09:06.000 --> 09:10.000 And this really helped us moving forward. 09:10.000 --> 09:13.000 So, what was the bug? 09:13.000 --> 09:21.000 In the Hocus Progress, the bug was that the documents were never getting unloaded out of memory. 09:21.000 --> 09:26.000 So, with time, the memory just kept getting loaded and loaded. 09:26.000 --> 09:30.000 And on the high load, a node just gave up. 09:30.000 --> 09:31.000 And why was this? 09:31.000 --> 09:34.000 That was because of the process dot next thing. 09:34.000 --> 09:38.000 And we were depending on a lot of event loop stuff. 09:38.000 --> 09:39.000 But yeah. 09:39.000 --> 09:42.000 That was eventually fixed and now things are good. 09:42.000 --> 09:47.000 Due to lag of time, I might need to skip this part. 09:47.000 --> 09:50.000 But yeah. 09:50.000 --> 09:52.000 This is how the client side code looks. 09:52.000 --> 09:56.000 Like these are the different states that we have on a client side. 09:56.000 --> 10:03.000 That helps us manage all the weird states that a client code go through. 10:03.000 --> 10:05.000 An able offline thing. 10:05.000 --> 10:11.000 And this force closed thing is something that is triggering a flow, force closed from the server. 10:11.000 --> 10:16.000 If the page gets too heavy or something. 10:16.000 --> 10:19.000 So, the local first UX. 10:19.000 --> 10:25.000 So, we have index DB ready on our report. 10:25.000 --> 10:28.000 That's what enables us to do. 10:28.000 --> 10:31.000 So, for first loading, I wanted to show a really cool demo. 10:31.000 --> 10:34.000 But well, it's a chaotic. 10:34.000 --> 10:37.000 So, this is how we do it. 10:37.000 --> 10:39.000 So, these are the different things. 10:39.000 --> 10:43.000 So, first we check if the server is synced. 10:43.000 --> 10:45.000 Then, yeah, show the content. 10:45.000 --> 10:47.000 Second, if say the server goes down. 10:47.000 --> 10:50.000 Like I just mentioned, then we are offline. 10:50.000 --> 10:52.000 Or if the user is offline. 10:52.000 --> 10:55.000 We show the cash content from the index DB immediately. 10:55.000 --> 10:58.000 And most importantly, if the cash is ready. 10:58.000 --> 11:03.000 And we know that we have a cash version. 11:03.000 --> 11:06.000 We show that immediately without any loading states. 11:06.000 --> 11:08.000 So, yeah. 11:08.000 --> 11:12.000 So, this is the flow that I just described. 11:12.000 --> 11:18.000 So, this was the super cool part that I wanted to share with you all. 11:18.000 --> 11:20.000 The server side editing bit. 11:20.000 --> 11:25.000 And this is not a detail we need to focus on. 11:25.000 --> 11:31.000 But sometimes you want server side editing as well along with your client side editing. 11:31.000 --> 11:34.000 So, say if you want to do operations. 11:34.000 --> 11:40.000 When the doc is not really loaded by any user, we need to do server side editing. 11:40.000 --> 11:44.000 But the challenge with server side editing is that you need to do. 11:44.000 --> 11:47.000 It's such that it doesn't override users changes. 11:47.000 --> 11:50.000 If they suddenly open the page somehow. 11:50.000 --> 11:59.000 So, a really neat use case is that say if you have a nested page sub documents basically. 11:59.000 --> 12:05.000 And if someone deletes the sub document, then the reference of the sub document from the main 12:05.000 --> 12:09.000 document has to be removed in the content. 12:09.000 --> 12:13.000 But to do that, you need server side editing. 12:13.000 --> 12:17.000 So, we follow the shadow wide of pattern that I came up with. 12:17.000 --> 12:23.000 And this also helps us to some cool AI stuff that I wanted to show. 12:23.000 --> 12:25.000 This is how it works. 12:25.000 --> 12:27.000 We create a shadow wide of. 12:27.000 --> 12:32.000 And we calculate the data between the live wide of and the shadow wide of. 12:32.000 --> 12:37.000 And then we apply those updates back. 12:37.000 --> 12:40.000 So, that now the updates are super safe. 12:41.000 --> 12:48.000 So, if you want to take any important points from this note, is these three points. 12:48.000 --> 12:52.000 And you should be good to go with your production stuff. 12:52.000 --> 12:54.000 And thank you. 12:54.000 --> 12:59.000 And let me just see if I can show you folks a quick demo. 12:59.000 --> 13:02.000 All right. 13:02.000 --> 13:07.000 Yeah, just a short demo. 13:07.000 --> 13:08.000 Is it fine? 13:09.000 --> 13:11.000 All right. 13:17.000 --> 13:18.000 Okay. 13:18.000 --> 13:33.000 Any questions? 13:33.000 --> 13:38.000 Any questions? 13:38.000 --> 13:48.000 Any questions? 13:48.000 --> 13:50.000 All right. 13:50.000 --> 13:53.000 Hopefully this was visible. 13:53.000 --> 13:55.000 Yeah. 13:55.000 --> 13:59.000 So, this was the server side thing that I was talking about. 13:59.000 --> 14:04.000 That enables us to do super cool stuff, even with AI and stuff. 14:04.000 --> 14:07.000 So, here you can see I'm asking it. 14:07.000 --> 14:11.000 Can you create a detailed blog on snapshotting in wages. 14:11.000 --> 14:17.000 And it starts to. 14:17.000 --> 14:19.000 The AI starts to generate content. 14:19.000 --> 14:22.000 I switch the page and work on something else. 14:22.000 --> 14:24.000 While it's doing it in the background. 14:24.000 --> 14:26.000 And it's a super cool like. 14:26.000 --> 14:29.000 And you can control it via an API call as well. 14:29.000 --> 14:32.000 So, you can trigger this exact same thing. 14:32.000 --> 14:36.000 So, you can create a API call like over here I make a curl request. 14:36.000 --> 14:37.000 Like this. 14:37.000 --> 14:40.000 Can you please remove all the headings. 14:40.000 --> 14:42.000 And it will do it in the background. 14:42.000 --> 14:45.000 And you can then accept or reject changes. 14:45.000 --> 14:46.000 But yeah. 14:46.000 --> 14:47.000 Like, server side editing. 14:47.000 --> 14:50.000 Really enables us to do a lot of cool stuff. 14:50.000 --> 14:51.000 Yeah. 14:51.000 --> 14:53.000 That's pretty much about it. 14:53.000 --> 14:55.000 As you can see. 14:55.000 --> 15:02.000 Thank you so much. 15:02.000 --> 15:04.000 If somebody needs to leave. 15:04.000 --> 15:08.000 Let's do it very quickly because we have people waiting to enter.