WEBVTT 00:00.000 --> 00:13.160 All right, hello, my name is Robin, and I'm joined today by my co-workers, Timo and Valer, 00:13.160 --> 00:14.640 we're from Elman's Voip team. 00:14.640 --> 00:19.440 We work on Elman's call mainly, and today we're here to talk to you about Matrix RTC, 00:19.440 --> 00:21.640 Excuto, a Battle Royale. 00:21.640 --> 00:27.120 Yeah, so of course Matrix is an open standard for decentralized communication, and it 00:27.120 --> 00:33.160 is divided into rooms, and in these rooms, users can send these persistent events to send 00:33.160 --> 00:36.960 text messages or things of that sort. 00:36.960 --> 00:39.960 But of course, this is just one kind of communication. 00:39.960 --> 00:43.620 There's a whole world of real-time communication that we'd also like to be able to do on 00:43.620 --> 00:51.360 Matrix, such as exchanging voice and video data, or maybe real-time data for a virtual 00:51.360 --> 00:55.280 world, or a whiteboard, or something like that. 00:55.280 --> 00:59.280 So yeah, what if we don't want this persistent event model? 00:59.280 --> 01:05.760 Well, we've spent the last few years trying to come up with a concept for Matrix RTC, 01:05.760 --> 01:15.240 which lets you augment your rooms with new features, so it lets you add new slots for 01:15.240 --> 01:18.000 real-time communication to your room. 01:18.000 --> 01:23.120 And so the possibilities here are that an admin can come in and add a slot for, I don't 01:23.120 --> 01:27.240 know, your Saturday night group calls, or your 3D virtual world, or your multiplayer 01:27.240 --> 01:32.200 crossword game, whatever you want, and so these all are things that get added to your 01:32.200 --> 01:33.200 room. 01:33.200 --> 01:40.400 And yeah, that is the idea behind a slot here, is that it is a combination of an application, 01:40.400 --> 01:46.160 which specifies what kind of data participants are going to exchange over this slot, and 01:46.160 --> 01:51.160 an identifying tag, just so you can have multiple parallel calls or something. 01:51.640 --> 01:56.200 Yeah, and so m.call is the very first application that we are proposing to add to this 01:56.200 --> 02:01.400 spec, and this is, of course, just the basic functionality that you need for any voice or 02:01.400 --> 02:06.280 video call, but third party apps are completely possible too, so you could just come in 02:06.280 --> 02:13.880 with a custom identifier here for your world simulation, and yeah, how this works is an 02:13.880 --> 02:19.000 admin will come along and they will send a state event, and this is how slots are recorded 02:19.080 --> 02:20.000 in matrix. 02:20.000 --> 02:25.320 There is stored in room state, so yeah, the admin here would send a state event with 02:25.320 --> 02:32.120 the application specifying the type, and there can also be some applications specific 02:32.120 --> 02:37.000 data there, and the reason why we are using state events for this is because it allows 02:37.000 --> 02:43.080 for matrix R to C slots to be moderated essentially, they can be opened and closed just 02:43.080 --> 02:49.960 by the admins, you can decide exactly what can happen in your room, yeah, and that way 02:49.960 --> 02:56.200 they are persistent and they are authorized, so yeah, the idea is that just like you join rooms 02:56.200 --> 03:00.520 when Allison Bob wants to get together on a call, they will connect to the slot within 03:00.520 --> 03:07.720 the room, and so how they do this is Alice is going to send a sticky event, and sticky events 03:07.800 --> 03:13.160 are another new concept that we are proposing to add to matrix, the deal with these is they 03:13.160 --> 03:21.160 are just like normal room events, except they have guaranteed reliable delivery for basically 03:21.160 --> 03:27.560 up to an hour, so whenever Bob comes along and sinks his client, he is going to receive 03:27.560 --> 03:34.440 Alice's sticky event even if it was like the scroll off the timeline, and yeah, so the nice 03:34.440 --> 03:39.800 properties of these sticky events is that they are a femoral, which is appropriate for 03:41.480 --> 03:48.760 these RTC session participants in a way that state isn't room state, it's like you can't 03:48.760 --> 03:53.880 delete an entry, so it sticks around forever, and they can be encrypted even, and they're also 03:53.880 --> 03:57.400 low privilege, you don't have to be an admin to send one of these because they're not a state event, 03:58.440 --> 04:02.840 yeah, and the most important thing here is that Alice's event declares how you can re-ter 04:02.840 --> 04:10.280 the RTC transports, yeah, and so how this works, well of course in reality Alice and Bob 04:10.280 --> 04:13.880 aren't a little Lego many figures sitting in a room together, they are people with computers 04:13.880 --> 04:20.360 sending data across in that work, and so the way that Alice and Bob end up exchanging their data 04:20.360 --> 04:25.720 is they are going to choose a transport, and the transport that we've now implemented for 04:25.720 --> 04:32.440 Ellen Call uses life kit media servers, but the idea is there could be more added to the 04:32.440 --> 04:38.680 specification for sure, and these are always going to be either resources provided by your home 04:38.680 --> 04:45.560 server, or maybe some peer-to-peer thing, so yeah each member will choose their own transport 04:45.560 --> 04:53.160 that's given by their home server, and then we let them, yeah, just upload their media there locally, 04:53.960 --> 04:59.320 they decide they're on transport because we want to maintain the federated model that we 04:59.320 --> 05:07.720 know and matrix even for real-time communication, and so yeah each member here will then go and 05:07.720 --> 05:15.320 also download the media from the others from MoT server, and so this seems maybe a little unusual 05:15.320 --> 05:20.120 for a calling system because you have to connect in multiple things just to participate, 05:20.120 --> 05:27.000 but the reason why I think we love this model and think it deserves to be in this 05:27.000 --> 05:32.360 fact is that it handles large calls gracefully without forcing them to be centralized, so you 05:32.360 --> 05:40.120 get a nice balance with scalability because the thing is with large calls, more often than not, 05:40.760 --> 05:46.680 many of the users will actually end up converging on the same home servers, that's just how large 05:46.760 --> 05:52.680 calls in like office environments end up working, there will be resources like a home server provided 05:52.680 --> 05:59.080 by the company that most users are on, and so they'll end up sharing transports, and that means 05:59.080 --> 06:05.320 that to connect to this larger crawl, well okay it's only of seven people, Bob only has to connect 06:05.320 --> 06:12.120 to just two servers to download the media here, and yeah so the live kit servers are still handling 06:12.120 --> 06:18.760 the fan out of all of everybody's media, which is the hard part normally for large calls, 06:18.760 --> 06:24.680 if you have to upload your media to and other people that gets pretty pretty expensive for 06:25.480 --> 06:31.560 phones that are on flaky mobile connections or someone on a DSL connection, 06:33.400 --> 06:39.800 yeah and so those are all the basics of matrix RTC already, so right there are four concepts you 06:39.880 --> 06:46.200 should remember, we have slots, this is how you set them up and moderate matrix RTC, 06:46.200 --> 06:51.320 the admin sets up the slot in the room by sending a state event, and that state event specifies 06:51.320 --> 06:59.000 an application, which is the type of data you're exchanging, and then to join, we connect by sending 06:59.000 --> 07:05.960 our membership event and we're publishing our media over our chosen transport, which is described there, 07:05.960 --> 07:13.320 so yeah slots, applications, memberships, and transports, now there are a number of other 07:13.320 --> 07:17.080 things that I haven't been able to describe here today because I don't have that much time, 07:17.720 --> 07:24.600 the entry is interesting stuff is the late events and notifications and encryption, but there are 07:24.600 --> 07:29.320 a number of other talks from last use matrix conference, if you're interested in learning about 07:29.320 --> 07:37.320 how matrix RTC works in detail, you can go check those out and yeah the MSC's for all of this are 07:37.320 --> 07:45.960 well yeah they're all up on the matrix spec proposals repository and I think they're not quite 07:45.960 --> 07:52.760 ready for this spec core team yet perhaps but yeah they're very close to being sent off, we're 07:52.760 --> 07:58.760 making our final adjustments to them and yeah so they all need your feedback, please go take a look 07:58.760 --> 08:07.880 at them and see what you think all right and we'll have to team up. Yeah thanks for the introduction 08:07.880 --> 08:13.480 or like the main part of the talk and thanks for giving us a background on the matrix RTC bits 08:13.480 --> 08:18.440 and now we at the point where we want to build something with it so with this like big changes 08:18.440 --> 08:23.400 or not big changes but this evolution of the spec we also did some really big changes to our 08:23.400 --> 08:29.480 code base to make it all compatible and have a proven implementation for the spec core team and why 08:29.480 --> 08:36.360 we did this yeah refactoring basically and adaptation to the new spec we noticed that we will 08:37.080 --> 08:42.360 eventually have a very clear cut on our code base which basically gives us something like a matrix 08:42.360 --> 08:49.960 RTC SDK for free and that's what we will talk about now so how to use it and what can you build with it 08:50.840 --> 08:56.680 so first this is like a very very rudimentary overview what our element code code base looks like 08:57.160 --> 09:03.080 we're building on the matrix JS SDK and those are all the parts which are just purely matrix 09:03.080 --> 09:09.080 related sticky events and state events and then we have the element code repository and this 09:09.720 --> 09:14.520 thanks to this refactoring and adapting everything to the new spec is now also pretty clearly split 09:14.520 --> 09:19.560 into two sections the bottom section is matrix RTC and the top section is that I'm really 09:19.560 --> 09:24.440 element call so the bottom section is things like connecting to life kid connecting to multiple 09:24.440 --> 09:30.200 SFUs because that's all not done on JS SDK and we noticed we can easily pull out this bottom 09:30.200 --> 09:37.320 block and expose it as a new built target so how does that look like basically here the sline is just 09:37.320 --> 09:43.400 in between the layer between the yellow and the green block and that is the API we decided for 09:43.400 --> 09:49.000 for this matrix RTC SDK so it's like pretty small fairly simple you have things like joining 09:49.000 --> 09:56.200 leaving tracking members as a observable so you see anybody any new joiner and lever 09:56.200 --> 10:01.000 and then of course you have the super important functions like sending real-time data, sending 10:01.000 --> 10:07.320 matrix data and inside the members you have the connections to them also yeah subscribe to the 10:07.320 --> 10:15.560 data and we even have like a high level data observable for all the life data and now what we 10:15.560 --> 10:22.200 are actually now going with the demo is that we use this SDK target and build like a very, 10:22.200 --> 10:30.360 very simple HTML template application for the good game and so you can use all the matrix RTC bits 10:30.360 --> 10:34.920 like connecting to life kid connecting to multiple SFUs handling delayed event handling sticky events 10:35.000 --> 10:41.480 authenticating with matrix all those bits extracted away and you just call await create matrix RTC SDK 10:41.480 --> 10:47.640 you get an SDK and then with the things we just showed you can say store the data observable 10:48.280 --> 10:55.640 into something called data um observable and then you can use that in godo so godo has like a pretty nice 10:57.000 --> 11:02.600 FFI basically for JavaScript so you use the JavaScript bridge get the window interface 11:03.560 --> 11:10.360 and then do dot matrix RTC SDK what we just stored in the window and then with this SDK we 11:10.360 --> 11:16.200 have access to our data observable and all the other things and but let's just look at the data 11:16.200 --> 11:23.400 observable one and we can use this in a game engine to build games with it and that's where 11:23.400 --> 11:29.000 pass over to Valera showing you hello so my name is Valera so I'm going to show you like the 11:29.080 --> 11:36.840 good at part so the part that probably will fail so yeah because we wanted to do to show like 11:36.840 --> 11:43.480 everything we do in matrix basically we build upon blocks building blocks but you can reuse to do 11:43.480 --> 11:49.080 most more things so this SDK events you could like send event that is but additional delivery 11:49.080 --> 11:54.280 guarantee super important to say I'm in the call we have the delay events so now you can do schedule 11:54.360 --> 12:01.000 event I can send this event this event tomorrow at 10 a.m. you can do self-destruct event but we want 12:01.000 --> 12:06.600 it to do a game so this is a full local setup here I'm going to do like that first we're going to try 12:06.600 --> 12:12.600 live after but you can see already that it's a very federation so they're not running on the same home 12:12.600 --> 12:19.880 server and so the first part is this is like this part is matrix communication so you can see the 12:19.880 --> 12:26.760 other side stay join game so I have now a game so here but everything that happened was in the room 12:26.760 --> 12:33.320 so I've got a aware of the membership now I'm going to do start but the start is going to send 12:33.320 --> 12:40.040 events by the data lifetime layer and I'm going to send the jumps so I can only jump for the left 12:40.040 --> 12:48.760 player so one of them will fall down so you should hopefully see me on the other side like you can 12:48.920 --> 12:56.360 see that I'm actually so this is like using two different life picture transport and now 12:57.400 --> 13:09.560 the matter of historic we created a public room I mean maybe a Timo and I would start a game so 13:09.560 --> 13:16.040 the room is here it's flat p dash or tc that falls down so I have to try from desktop I don't think 13:16.040 --> 13:22.120 you can from mobile it's now which it's support yet but if you can we will let this room 13:22.120 --> 13:30.440 open a bit if you want to try a bit later let's let me first start the game some waiting for players 13:32.280 --> 13:35.000 so that's the part where we see if we have good network 13:35.000 --> 13:51.800 it's like you stuck so okay I have a demo I mean I've recorded something so it's like 13:52.600 --> 13:57.640 when we started that I say how hard can it be to have a world-but world-back network code 13:57.640 --> 14:02.680 because I wanted it to be like really real time I wanted the birds to fire bullets 14:02.760 --> 14:07.560 things that make you do small jumps so I said what a world-but-net will hurt it's easy 14:07.560 --> 14:12.280 but here it was with a bad network and you will see that the other side is like 14:12.280 --> 14:18.680 world-back and catching up so teleporting but yeah that's how it looks like with two people you know 14:21.400 --> 14:27.240 so let me go back and see oh it's three players right okay that's the time 14:28.120 --> 14:41.320 four let's try it oh which one I mean okay okay so if there's too much like it's going to last 14:41.320 --> 14:46.840 the players I'm dead 14:46.840 --> 14:55.800 so yeah I've had basically the game so you can add that so I think I would push about 14:55.800 --> 15:00.840 to get to other people want to try out in some way a room if you want to have a break 15:03.720 --> 15:05.000 I don't know who he's top one 15:07.320 --> 15:12.600 are those five players so they don't want to play or more so there's also another game I just want to 15:12.600 --> 15:19.720 show you so it's a widget so basically I can go here next tension we have another game I won't be 15:19.720 --> 15:24.760 able to type but because I'm holding the microphone but just to show you that now it's 15:24.760 --> 15:32.040 we'll ask you for a wall of permission it's a bit scary so like the widget want to take control 15:32.040 --> 15:41.640 of your life so it should be loading or maybe I'll already load it oh this is the wall okay 15:42.280 --> 15:46.280 so there's a lot of legacy haven't here it's long so but something we need to work out 15:46.280 --> 15:55.320 in the widget API to rescue a bit to group a bit to permission service and yeah is this 15:55.320 --> 16:03.240 single last bug it's requesting that several times okay yeah you can yeah we do have a couple 16:03.240 --> 16:09.720 players though I can I get rid of my team oh are you just have to press it as often as it 16:09.800 --> 16:14.920 called it okay there we go so here the game is like you have to type or I have to do something 16:14.920 --> 16:20.760 you have to type to prove or you have to start the game right I think it started yeah so everybody just 16:20.760 --> 16:25.240 types if they if they are not sure what exactly they can look at the three-letter preview on the 16:25.240 --> 16:30.520 button left corner and then we have multiple people on the track I think for the layer they 16:30.600 --> 16:37.560 I don't know I'm a little I'm a computer I show as little good old icons self of course one 16:37.560 --> 16:46.360 I've been at to fail but as soon as one person will will make the full rounds it will send 16:46.360 --> 16:50.760 into the room how quick they run it looks a lot better on this one yeah should we just like 16:51.720 --> 17:02.680 that's time we try to play I can try to reload it oh yeah I think I already I mean they 17:02.680 --> 17:09.160 bit and been this one maybe it doesn't like it and they're out two of them I think I already clicked 17:09.160 --> 17:18.840 in that oh I can see people now oh they are okay okay how much I have to click and 17:19.480 --> 17:25.160 oh I'm not clicking oh yeah okay so yeah people are racing you race by typing the letters 17:27.560 --> 17:36.920 I'm sure I'd you go you could have put some real words yeah so that's it so like this is 17:36.920 --> 17:42.040 a bit on top of the matrix RTC just to show like all the things you can build with the SDK 17:49.480 --> 17:52.760 so any questions 18:01.320 --> 18:05.120 and when I only copy I just basically gave you a thing and having all just the 18:05.120 --> 18:10.200 key features just for the fake one to make all the games all the time we just put 18:10.200 --> 18:15.560 up we know and use this real time for back in that I have and can use it I guess I have to 18:15.640 --> 18:24.840 use it roughly you have to do that sort of RDP the question so basically if you want to 18:24.840 --> 18:29.880 do a game but using something else that good up yeah another language so you just have to 18:29.880 --> 18:39.000 have that sort of bridge so as we say in God it's quite easy like I have a JavaScript bridge 18:39.640 --> 18:42.600 I mean just always meet you get a 18:43.400 --> 18:48.120 I feel people can see things but I'm just getting the bridge I'm getting the windows and I'm 18:48.120 --> 18:52.120 getting the match-off season if you have that bridge then it's just Mario of listening to heaven 18:58.120 --> 19:00.920 oh you can do the game in JavaScript and I think we'll just work 19:09.000 --> 19:22.920 and we can use and we can use it you can't use any of the GS SDK you have to build some 19:22.920 --> 19:27.720 income and some benefits and I think there's another dimension or yeah I think there's another 19:27.720 --> 19:32.760 dimension to this question so basically the way we develop RTC applications currently is 19:32.760 --> 19:37.800 always as a widget that has a big advantage that you don't need to give some random game 19:37.800 --> 19:43.480 your full matrix account with all the encryption keys but instead element web or whatever other 19:43.480 --> 19:50.120 trusted widget supporting client you have will do all the encryption and key back up and everything 19:50.120 --> 19:54.920 and the widget can just talk but it really needs to talk with this client and as soon as you 19:54.920 --> 20:00.840 do a widget you're forced to use an iPhone anyways so the alternative would be really building a full 20:00.920 --> 20:07.080 go client with then also a go-based matrix RTC SDK so then you cannot really benefit 20:07.080 --> 20:13.000 from this work it's really for most the beneficial for widget developers and yeah our 20:13.000 --> 20:18.520 idea is also that this is basically the safest and most the way to do RTC applications because you 20:18.520 --> 20:24.280 get all the security benefits from not leaking your full key back up to the RTC application 20:25.240 --> 20:34.040 yeah oh well it's the first thing so it's right and it's understood the transport part is also 20:34.040 --> 20:38.440 structured and it's the thing on the signaling and managing the slots you have to do so if I 20:38.440 --> 20:45.880 were to use some other transport like that so I can protect it in a place to implement the project 20:46.680 --> 20:53.000 yeah exactly so basically the SDK currently is very much focused on um real-time applications 20:53.000 --> 20:59.080 not really voice-in video since you have multi like multi widget support per room the idea is also 20:59.080 --> 21:04.680 that there will be dedicated m.col implementation that do your calling and if you want an experience 21:04.680 --> 21:09.560 like talking to people while playing a game you would just run two RTC applications at the same 21:09.560 --> 21:14.680 time and yeah to keep the SDK simple it really is just focus on the data transport 21:15.560 --> 21:21.400 and then yeah if you have a different kind of transport it would all be abstracted away in the RTC SDK 21:21.400 --> 21:27.640 and yeah you could use it okay yeah we just learned in the previous part that's a RTC case 21:27.640 --> 21:40.680 so much better and the idea of mine to do from the RTC as the K to RAS or yeah yeah I can also 21:40.680 --> 21:45.800 do a comment on this maybe yeah we also hear other comments so basically as I said before 21:45.880 --> 21:51.480 most of matrix RTC currently is built for the widget target because that makes a lot of sense 21:51.480 --> 21:57.800 for the current use cases and maybe also for a couple of future use cases and of course 21:57.800 --> 22:05.320 the best way to use a matrix RTC application is to have a RAS SDK client exposing a widget API 22:05.320 --> 22:11.560 over the RAS SDK and then just having your actual RTC application running as a JavaScript widget 22:12.440 --> 22:18.440 otherwise you would need to build like or there's discussions for building modules of the RTC 22:18.440 --> 22:25.560 logic in RAS and then run it as web assembly in the widget and so I think we really talk about 22:25.560 --> 22:32.920 like a full-fledged RAS based matrix RTC SDK as soon as people start to build native implementations 22:32.920 --> 22:39.000 of voice and video calling which are not eye-frame based maybe somebody else wants to add something 22:39.000 --> 22:47.400 to it yeah I would just add that let's see here I would just add that the dependency that like 22:47.400 --> 22:55.480 element call for example currently has on matrix JS SDK it really doesn't need to be using 22:55.480 --> 23:01.800 JS SDK at all because yeah it is ultimately just making widget API requests rather than client 23:01.800 --> 23:08.680 server API requests and so the only reason we've gone for JS SDK there is I don't know we've sort of 23:08.680 --> 23:14.680 hacked it so that it redirects things over the widget API and just has a familiar sort of JavaScript 23:14.680 --> 23:21.880 interface that lets us develop element call for widgets and this like standalone SPA mode at the same 23:21.880 --> 23:29.480 time but I don't think we want to have the SPA indefinitely at least like once we get all the 23:29.480 --> 23:35.720 features of the SPA working just within our element messenger clients so that you can even like 23:35.720 --> 23:42.520 to yes access through them at that point we would probably be able to switch away from the JS SDK 23:43.320 --> 23:56.360 and have something more custom made for widgets yeah anything else no okay thank you for coming to the 23:56.360 --> 23:58.360 talk