The Matrix Space Beta
matrix.orgFor what it’s worth, the thing I find most exciting about Spaces is that they provide a decentralised hierarchical namespace with decentralised access controls for every room (ie pubsub topic) in Matrix. So it’s like we’ve sprouted an openly federated global hierarchical filing system for freeform realtime data streams of all flavours - where people can go crazy defining their own trees, applying their own curation ideals; perhaps we’ll even see a single global tree emerge (although the implementation may need some more optimisation first).
It’s like a multiplayer hybrid of DMOZ and USENET and the read/write Web all rolled together. Once we start storing more interesting data streams than instant messages in it (eg forums, email, bulletin boards, DOMs, scene graphs, ticker data, IOT sensor data...) it really gets interesting :)
Plus you can use it to organise your own rooms and have Discord style communities or Slack style workspaces, but that’s the boring obvious bit ;)
Edit: for a user-facing rather than developer-facing overview, https://element.io/blog/spaces-the-next-frontier/ has more details.
I wish Matrix would instead focus on fixing all the existing encryption usability issues instead of these new features:
https://github.com/vector-im/element-web/issues?q=label%3AA-...
I would love to migrate my family to Element (and also friends, and eventually also recommend it to employers if they're ever choosing something other than Teams) but can't until it's reliable as webmail. I also wrote this comment:
https://news.ycombinator.com/item?id=25271512 "Once Element is mature enough (and I'm sorry, but looking at the incoming issues on https://github.com/vector-im/element-web/issues?q=sort%3Aupd..., it doesn't look like it yet), then hopefully more companies will start considering it."
I get that E2E means it cannot be as simple as email, and I can handle the extra training that's needed, but not for figuring out all these known issues and attempting to prevent people hitting them, or having to handle the situation if they do.
So there is a finite number of active contributors to Element, and we have to prioritise what to work on. While E2EE is not perfect (the UX needs another major iteration and we are still hunting rare edge cases where messages cannot be decrypted) in practice it's usable day to day.
We consciously chose to prioritise building out Spaces over the last few months over E2EE UX as otherwise there's a risk of Discord becoming the de-facto home for open source projects, much as Slack started to be a few years ago - which would be catastrophic for open standards-based communication like Matrix.
Meanwhile, the cryptography team chose to focus primarily on implementing next-generation encryption (IETF's MLS) on Matrix rather than polishing the current behaviour - given MLS should both radically improve scalability, but also fix the majority of the edge conditions which are problematic for today's E2EE, or at least entirely switch bugs in the existing implementation for entirely different failure modes in the new implementation. We showed off MLS over Matrix last week (https://youtu.be/xn0fzyimycs?t=248), and we're now finishing the decentralisation component of it (https://matrix.uhoreg.ca/mls/ordering.html).
Eitherway, now that the Spaces beta is out the door, we're catching up on other UX issues, including E2EE. We also have more folks being paid fulltime by Element to work on encryption (amongst other stuff) starting in July. Talking of which, if anyone wants to get paid to make this happen sooner, Element is hiring at https://apply.workable.com/elementio.
Thanks for the response, that sounds good and makes sense, I guess I'll continue waiting...
My family regularly does things like forget which browser they last used, or re-install windows, or otherwise mysteriously have all browser settings disappear, etc - all things that seem to be related to those key-backup issues.
Put it on their phone or tablet.
PCs are no longer the right device for "mom and dad".
I'll just say; both of these seem like buckets of good priorities. I too wish some aspects of the onboarding were simpler, but I also am very very excited about Spaces, and have a number of communities I'm stoked to onboard once that feature launches.
Kudos to you and the team and I've been really enjoying tracking your progress over the last ~4 years.
Enabling by default:
- cross-signing (so users verify their own devices themselves, and you verify users only once by verifying their public key, regardless of how many times they add or update devices)
- and key backup (so moving between devices doesn't need manual polling for other devices' keys the first time for decrypting messages)
would be a great step forward. Those are there, but they are disabled by default yet, which is a disservice.
cross-signing & key backup are enabled by default, and have been for a year or so? you have to explicitly opt out by repeatedly hitting the 'skip' button. however, the UX is certainly more complicated than it should be, and we're due to rework it in the coming months.
Woops. I may be dealing with friends & family that joined prior to that. I wonder if one could get prompted again for enabling cross-signing & backup once you open a new client (I don't think it happens right now). That would help migrating everyone to enabling it.
I'd love to see those encryption usability issues fixed, but I do want to say that e2ee on Matrix is less bad than on anything comparable (like OMEMO on XMPP). I like XMPP because of its speed, but you really can't combine OMEMO and multi-device unless you just don't mind having parts of your message history unreadable on some devices. The key backup and session validation in Matrix/Element usually solves this issue.
Unfortunately, the complexity means that even other Matrix clients that support e2ee don't support key backup and session validation, which means you either stick to Element on every platform, or live with the same history issues as on XMPP.
The iOS client is miles behind their Android client too.
> On iOS, we're still busy implementing Spaces. However if you join rooms which belong to Spaces, you’ll still be able to talk in them.
E2EE is opt-in per-room.
The entire point of Matrix is that it has E2E. If E2E isn't a requirement then my prefered communication tool is email.
> Discord style communities
Boring and obvious, but this is probably what most people would want. Now that we've seen how good and usable the Discord model is, it's pretty much the baseline now for public chat rooms.
It sucks there's no open source project that mimics discord well.
They all shoot themselves in the foot by over complicating the model. It confuses users, and makes creating stable clients hard.
I don't want a tool-box of primitives and a recipe book. I want a decentralized discord clone not owned by a giant company.
Let's start there and add the crazy stuff later.
To build an open source decentralised Discord alternative you need a tool-box of primitives - that’s what Matrix is. Then clients like Element (https://element.io) sit on top to fill in the rest.
We don’t think the model for Spaces is overcomplicated. In fact it’s simpler than Discord (no role-based permissions; just power levels).
Is there any documentation detailing power levels, or that decision was made to only offer power levels but not role-based permissions? FWIW, the reason most major platforms offer role-based permissions is because enterprises demand it, and it’s become the de facto primitive for authorization. User type based permissions also make sense, but are often used in conjunction with (not in lieu of) roles.
Power levels feels like the most IRC answer possible, and breaks down as soon you want mutually exclusive abilities.
Why is this the preferred model?
Because the decentralised ACL implementation (state resolution) relies on power levels as a simple comparator for determining merge conflict resolution. State resolution is the academically novel bit of Matrix and hard to get right, and while we can swap it out for role based ACLs in future, this is orthogonal to Spaces. Meanwhile it honestly hasn’t come up as a requirement, even from enterprisey folks, given we push power levels as hard as possible.
https://matrix.org/docs/guides/implementing-stateres and https://matrix.org/blog/2020/06/16/matrix-decomposition-an-i... give more info on how state res works and why you wouldn’t rush to jump to role based ACLs.
It’s nothing whatsoever to do with IRC or chanserv style behaviour, beyond coincidence.
I should also add that there are at least two matrix spec change proposals flying around to add role based ACLs, and one of the main reasons for switching spaces to be built on rooms is that they will automatically inherit the ACL model when it lands :)
And neither of those two proposals have figured out how to solve the mess of the conflict resolution.
Solving conflict resolution should be doable - it's just that it's literally the most critical and important bit of the entire Matrix ecosystem, and any changes have to be done very deliberately and carefully, ideally with formal proofs that it's not going to break stuff. It's the equivalent of switching out Git's merge algorithm (https://github.com/git/git/blob/master/Documentation/technic...) or switching out for a new kernel scheduler implementation. Any bugs or thinkos there mean that the whole fabric falls apart, hence wanting to consider it separately to Spaces.
Great response, thanks!
It’s funny you mention that — I just started such a project (three hosting modes: centralized, federated, fully decentralized) as an educational series to teach software development on multiple levels. It’s very early (read: non functional), and I’m still figuring out the best way to teach and build simultaneously, but constant weekly progress is being made.
Educational site (I’m backporting the previous sessions to the site, session 7 is the first one hosted right now): https://www.codewolfpack.com
Project repo: https://www.github.com/HowlerChat/Howler
There is an attempt named harmony with multiple clients. https://github.com/harmony-development
Disclaimer: one of my friend is involved in this project
yup, and to be clear, that’s why we built Spaces. doesn’t stop the “wow we accidentally created the realtime read/write web” thing being cool tho ;P
Wohoo! Thanks to Spaces, I feel comfortable using Cactus comments for blog posts (e.g: https://viccuad.me/blog/enabling-cactus-comments), as I could create a private space containing all the rooms for each blog post comment chain.
To me it shows the snowballing effect of features.
Also, Matrix's Spaces (even in Beta) seem more flexible and thought out than the second best implementation I know, in Slack.
Definitely check out Zulip for a pretty sweet threaded model also: https://news.ycombinator.com/item?id=27149123
Matrix’s threading model has been designed to support Zulip (we’ve chatted a bunch to Tim about linking them up) - the only catch is that it’s not implemented in any clients other than Cerulean so far. Element should add support soon now Spaces are out though.
What's the Matrix analog for Zulip? My understanding of Zulip is that you have an instance, streams in that instance, and topics in that stream.
Maybe in Matrix the hierarchy would be implemented by nested spaces? A space for the stream and one for each topic? That would probably work if not for the critical Zulip feature of being able to move messages between topics (and between streams?) to better organize the conversation.
Maybe one Zulip instance is a giant Matrix room and the threading model that's landing soon is used to implement the relationships? I know the upcoming threading model can dynamically change event relationships. But then maybe it would be better to avoid giving single rooms that much traffic.
There are probably lots of ways to implement that functionality. I'm curious as to what you were thinking.
https://matrix.org/blog/2020/12/18/introducing-cerulean has some details - and https://github.com/matrix-org/matrix-doc/pull/2326 and https://github.com/matrix-org/matrix-doc/pull/2836 are the two spec proposals underlying it.
Obligatory comment here to say that Zulip is amazing.
Based on your comment, I tried this and it is working. Thanks.
Are you adding the each blog post comment room to a private space manually or is there way to automatically add it to that space?
What is the E2EE story for spaces? Besides space names and member lists which I assume are visible to the server like normal rooms, what other information is also visible to the home server? Are rooms/subspaces inside spaces also visible?
Spaces are identical to rooms, which means that state events (key value data like name, membership, avatar, etc) are currently not e2ee, but conversation data would be e2ee if there was any. Encrypting human visible payloads for state events is on the roadmap however, but encrypting the metadata would be hard. For that, we are working on P2P Matrix to stop the metadata pooling anywhere.
> For that, we are working on P2P Matrix to stop the metadata pooling anywhere.
For everyone who just glanced through the comment above, this is big and a major selling point compared to most competitors (IMO).
It's also early beta, though (although we're committed to getting Matrix hybrid client-server & p2p in the medium/long term). https://matrix.org/blog/2021/05/06/introducing-the-pinecone-... has more on the latest developments on P2P (and public beta builds for iOS & Android) from a few weeks ago :)
Pretty exciting stuff, still trying to wrap my head around some of the technical possibilities outside of chat. Just to be clear, a room can be contained in multiple spaces, right? Or even, if I'm understanding it correctly, in multiple places in a hierarchy in the same space?
The Element blog post talks about setting up admin spaces and community spaces. So if rooms aren't exclusive to a single space I could build custom, private spaces that pull rooms from multiple different communities I'm a part of.
That's a really cool feature that Discord doesn't have right now.
There are lots of different things you can do with Spaces. But yes, you can build your own Space out of arbitrary rooms you don't own without anyone's permission.
Then there are what you might call "canonical" spaces, where space A labels space B as its child, and B labels A as its parent. Then you know that the relationship between the two is mutual, and clients can take that into account.
But you're free to create your own Spaces that are the parents of other rooms, where those rooms/spaces are unaware of the Space you created.
This was possible in the old Matrix community system as well, it acted more like a "filter" than a community though.
Once this becomes widespread, I can actually start using Matrix as a real Discord / Slack alternative (communities just weren't cutting it, unfortunately).
Once the audio chat feature is seamlessly integrated, I even consider ditching my old mumble server for it. Or not. Probably not. Our group just needs audio, with the occasional picture flood, until they get it out of their system. Since some (me include) won't use Discord due to privacy concerns, I still hope for a good and reliable self hosted alternative. A more modern user interface sure would be nice.
voice/video rooms are planned for June.
Great to hear! I'd love to be able to recommend Matrix/Element as a Discord alternative.
However, actually competing with Discord will require a tonne of hard work in Element. The little things like automatic voice activation, echo cancellation and volume normalization are the reasons Discord are as fantastic as it is, so Element will need to be excellent with those to compete.
The first cut will be using Jitsi as the engine, albeit with more appropriate UI, which gives you AEC (echo cancellation) from WebRTC and some level of AGC (normalisation). But we have plans to go far beyond that, and are very aware it’s hard work. However, pre-Matrix, the core team professionally built VoIP stacks, so we have some experience here (and our own WebRTC implementation should we need it :)
I would love to see you use Jitsi and contribute back, where possible.
Also, if you move on for reasons that aren't bike-shedding then I'd love to know what the architectural/technical reasoning is.
The main reason to consider something different to Jitsi is to directly use Matrix for decentralised e2ee signalling to manage the media streams, and allow hybrid SFU and MCU models (like hangouts or zoom) rather than pure SFU like Jitsi. We do like Jitsi though and already contribute directly and indirectly - Jitsi’s E2EE is derived from Matrix, and the Matrix community just contributed a tonne of a11y fixes to it that just landed. But we’d still like a fully decentralised Matrix-native group call solution eventually.
Wonderful! I think you've answered any questions I might have already.
If you do AGC, you'll already be more usable than Discord, who apparently refuse to implement this. Whenever we're voice chatting in my group, it's extremely annoying that one person is quiet as a whisper and the other is AT 200% VOLUME AND CLIPPING.
The risk on AGC is that unless combined with good voice activation you can end up amplifying background noise and then deafening people when they start speaking. But yup, we'd definitely want to do this.
Oh, sorry, you meant input normalization, of course. Please please please also add output normalization, so all people in a room sound at the same volume!
I think it depends on which perspective your inputs/outputs are coming from ;) The way for all people in a room to sound at the same volume is to normalise the audio you capture from them. If you normalise the mixed signal you play back to the other users in the room, it's too late (at least if they speak over the top of each other), plus it'd be too hard as you'd be constantly yoyoing around.
Oh hmm, that's true, thanks for the clarification.
> automatic voice activation
Voice activation is a client side thing.
Yes, and Arathorn works on element, does he not?
he is one of the founders
That's amazing! This is such a smash-hit feature from Discord that no other messenger has gotten right yet.
Feels like a needed step for community-building after having used Discord for so long. You are doing awesome work.
Can someone remember to post this to HN when it happens I will definitely forget
Cool! Is there a way to try the feature already? Can I just install the development versions of synapse and element?
Many thanks for the work you do!
This has been the main thing preventing me from switching to matrix full time, very exciting!
I would love to ditch slack/discord for matrix. Unfortunately many communities are reluctant to maintain a bridge
So I understand why the spaces api is similar to rooms from a developers perspective, but why do they share the same # sigil as rooms and not take the old + community sigil?
Spaces seem conceptually different from rooms.
Under the hood, spaces are rooms - and we made the decision to keep using a # sigil to refer to them, given the act of joining and being invited and having multiple aliases etc is the same as a room. When you say “hey, come join #community:matrix.org” does it really matter if the thing you are joining is a room or a set of rooms?
That said, it’s subjective, and we went back and forth on it. My original proposal had a twigil of #+wherever:example.com, which in retrospect was daft. I think it was a member of the community who proposed just using plain # aliases - hard to tell given https://github.com/matrix-org/matrix-doc/pull/1772 is so heavily commented that it crashes if you expand the comments :/
Sorunume suggested just + then later Alch-Emi made the case for just #.
https://github.com/matrix-org/matrix-doc/pull/1772#discussio...
I agree. There has been some discussion regarding this and I think it's likely Spaces will get their own sigil (but not sure whether it will be +).
This is really cool tech. I'm using it in my "Discord meets Minecraft" app (jel.app), it's basically exactly the primitive missing to really have a multi-tenant story for these kinds of chat systems.
I looked at this post and the user facing blog post, but I’m not sure if I’ve understood this. Is it just a way to organize different rooms together for easy access (like Telegram’s folders, which is just for each person, but here these can also be shared with others)? If yes, then can someone please point to similar implementations (preferably with screenshots) on other platforms for comparison and where the similarities and differences lie?
discord servers/guilds, slack teams
Anyone here using the Matrix ecosystem instead of Teams for a more "enterprisey" company?
At Element we have a bunch of customers doing so, but the one I can speak to most accurately is Element itself, as a company. Our internal space view looks like this, for instance: https://matrix-client.matrix.org/_matrix/media/r0/download/m.... As you can see, it's pretty enterprisey - we even have HR rooms ;P
You have forgot sanitising the screenshot before sharing with us.
Nope, I deliberately shared it unsanitised to try to demonstrate a real life enterprisey use case. There's nothing sensitive shown.
A good and robust bot api and useful smart widgets based on url are a must. I've had chats go from discord to matrix and then simply die after a few days.
Matrix's bot support should be excellent? It's arguably more robust than Discord's given it's an open API and there are tonnes of SDKs and bots building on it: https://github.com/turt2live/matrix-bot-sdk, https://matrix.org/docs/guides/usage-of-matrix-bot-sdk, https://github.com/bnjbvr/botzilla, etc.
What do you mean by 'smart widgets based on url'?
Things like Spotify links turning into an embed widget.
I've used the bots api for slack and discord in a corporate environment. Compared to that the matrix api is horribly behind and unreliable.
thanks; filed https://github.com/vector-im/element-web/issues/17396 for widgets.
Not sure what problems you're seeing with the Matrix API; it's lightyears ahead of Slack & Discord in my experience, but I guess i may be getting high on my own supply...
> As many know, over the years we've experimented with how to let users locate and curate sets of users and rooms in Matrix. Back in Nov 2017 we added 'groups' (aka 'communities') as a custom mechanism for this - introducing identifiers beginning with a + symbol to represent sets of rooms and users, like +matrix:matrix.org.
We already have a universal notation for identifying resources on the Web. Why not use that?
That would be https://github.com/matrix-org/matrix-doc/pull/2312, which is merged and registered with IANA and coming to a web browser near you real soon now.
...because it's a different protocol with different needs that are satisfied by its own notation?
Edit: Just read Arathorn's comment above, very cool - I hadn't seen that one before.
This is very exciting! Been waiting for this for a while
Finally!
Do the Matrix clients still phone-home to some centralized server (Vector, I think it's called) as well as to matrix.org on every launch?
That's what kept me from switching last time; to me a federation client that's going to phone home to a central server is no better from a privacy standpoint than using Signal (which isn't federated but is also end to end encrypted and phones home to its centralized server), and Signal's UX is better thus far.
If that can get solved (and it seems like it can), Matrix looks like it's going to be super awesome.
> Do the Matrix clients still phone-home to some centralized server (Vector, I think it's called)
There's no centralized server. There's a server with the majority of users for now (in matrix.org), but the network is decentralized, with many servers. The server implementation run by the majority is called Synapse, but there are others (Dentrite, Conduit, Construct..).
I'm puzzled by this comment. Can you elaborate on what you mean by phoning home?
Are you talking about https://element.io, which points to matrix.org homeserver and signing server, etc?
In that case, well, just use a different server and your own client. Or self-host, and select one of many clients: https://matrix.org/clients/. That's the whole point of federation.
You can even deploy your own web client pointing to whichever server default you want, many people do. E.g: element.debian.social, chat.mozilla.org, etc.
> I'm puzzled by this comment. Can you elaborate on what you mean by phoning home?
On first launch, it connects to multiple centralized servers before you have a chance to select your homeserver.
The issue was with the default client, in the default config.
Not sure if it's still happening, but it was concerning enough that it happened in the first place.
https://github.com/vector-im/element-web/issues/13942
https://github.com/vector-im/element-web/issues/12712
https://github.com/vector-im/element-web/issues/11655
https://github.com/vector-im/element-web/issues/11655#issuec...
It would also appear that it intends to maintain a Solarwinds-style remote code execution vulnerability on the machine on which it is installed:
https://github.com/vector-im/element-web/issues/11655#issuec...
As per my comment at https://github.com/vector-im/element-web/issues/11655#issuec..., we'll switch auto-update checks to require user confirmation.
Did you maybe post the wrong link?
The comment you're linking to says that there are no current plans to change the default request to matrix.org from the login page, and that you explicitly plan to keep the current auto-updating behavior.
(also, of course "we will fix that" implies that it is currently still broken, which is what the original commenter was pointing out)
The comment i linked is me saying:
> we're going to fix this, and yes, imo, autoupdates need an opt-in.
i'm not sure how clearer i can be.
The comment you linked says:
> As should be pretty clear from #11655 (comment), we're going to fix this, and yes, imo, autoupdates need an opt-in.
The comment #11655 says:
> Auto-updating for default Riot distributions will continue as before to ensure security fixes are delivered quickly
So at this point already it becomes unclear to the reader which of these has precedence.
So, assuming you're asking in good faith, if you wanted to be clearer you should have written something like:
"*Despite* what's written #11655 (comment), I disagree with the product teams assessment and think autoupdates need an opt-in, therefore I will make sure this is implemented."
(not sure if the content is 100% what you wanted to express in the comment, but you get the idea)
When can I install and run a matrix client without it connecting to you? I want to use your software but want to self-host it and not use any services whatsoever from your organization.
I suspect they're referring to https://matrix.org/blog/2019/09/27/privacy-improvements-in-s... and https://matrix.org/blog/2019/06/30/tightening-up-privacy-in-...
They could be referring to the vector identity server for email/phone number -> matrix matching. I believe this is enabled by default on synapse installations (self-hosted ones too), but the URL can be changed in the config.
The relevant thing is whether the client enables it by default. From my experience, Element iOS doesn't.