> With XMPP and Matrix.org -based services you would still need to convince everyone to join your new network. Easy in theory, very complex in practice!
I think they missed the the bit where Matrix is called Matrix because it bridges (matrixes) the existing networks (Slack, IRC, Telegram, Discord, XMPP, etc) in, rather than needing to convince everyone to join. But no matter, we'll just provide a COI bridge if this takes off :)
> "Jabber is a new project I recently started to create a complete open-source platform for Instant Messaging with transparent communication to other IM systems(ICQ, AIM, etc).
Matthew Hodgson, 2019:
> I think they missed the the bit where Matrix is called Matrix because it bridges (matrixes) the existing networks (Slack, IRC, Telegram, Discord, XMPP, etc) in, rather than needing to convince everyone to join.
Those are some really good examples of the opposite of what you intended to say.
> * Wonky unwieldy hypertext systems instead of the WWW
Which ones? Gopher wasn't hypertext, but it is still going and still a good idea. A conversion to Gopher would be a huge usability and accessibility upgrade for many current websites.
> * The Nomad instead of the iPod
iTunes is garbage. iPods sucked until (certain models) had Rockbox ported to them.
> * The Blackberry instead of the iPhone
How about the Nokia n900 instead?
> * GNU Hurd instead of Linux
I'm "stuck" with BSD. Send help?
Ambitious projects are cool and should be encouraged. Being an ignoramus about existing developments (not saying that that is what the people behind Matrix are doing) and getting distracted by anything that claims novelty should not be. Hoare had a great bon mot about this: "Here is a language [Algol 60] so far ahead of its time, that it was not only an improvement on its predecessors, but also on nearly all its successors."
> The products you mention are all good examples of niche products for techies.
The blind person that told me about how much better Gopher was than the modern web for blind people was not a techie, and I don't think it is moral to treat disabilities as just a "niche."
I don't ever remember being 'stuck' with GNU Hurd. It was a "non-product" with a little more evidence of progress than Duke Nukem Forever. Back in those days, I used SunOS (later Solaris), HP-UX, DEC Ultrix, IBM AIX, Tandem Guardian and SGI Irix.
The more likely outcome is that GNU Hurd would be developed to be similar to Linux if Linus chose to stick with existing projects instead.
Linux had real improvements over Hurd but it'd be silly to suggest that a separate "Linux" brand kernel needed to exist to make those improvements possible.
That's not how interconnects work. You don't even have to connect to Matrix, you can just use it to link existing systems more easily. Pretend there is no protocol/standard and it still does a good job.
In reality, those bridges work badly and hardly anyone uses them. I had to give up on my attempt at using Matrix as IRC client - it had questionable uptime and some channels even banned Matrix users due to frequent reconnects (on Freenode).
Hackint recently shut down their Matrix bridge due to how maintenance-intensive it was (and the fact that it logged all messages in its own database).
This info is a bit stale - it's true we had stability problems in Freenode until July 2018 (exasperated by their spam issues), but since then it's been pretty stable and the reconnects are considerably rarer than freenode netsplits. It's not perfect, but many people use it fine as a casual IRC client. There are over 24,000 active users using the Freenode bridge in the last month, for instance, so I'm unsure that 'hardly anyone uses them'...
The Hackint story is a different one; the bridge itself is not maintenance intensive at all. But it is true that the default Matrix server implementation is resource & relatively admin heavy; we're working on that currently. Meanwhile we're also fixing the data retention behaviour - https://github.com/matrix-org/matrix-doc/blob/matthew/msc176... etc.
I tried Matrix last month. It's neat, but slow and a little buggy. Recommending it to anyone other than a Matrix developer right now seems a bit of a stretch.
Not to mention that it's just re-solving all the same problems other messaging protocols (XMPP, IRC, etc.) spent time solving 10+ years ago.
I always feel that the Matrix organization is a bit disingenuous. Not only are they not developed by any recognized standards body (meaning they have no long-term sustainable funding model), but if they really just wanted to connect all the other chat protocols and make sure everything was interoperable they could have built bridges for existing protocols instead of reinventing the wheel yet again and making the chat scene even more fragmented. Not invented here syndrome is not okay when there are plenty of good (or at least "good enough" even if they're not my favorite) alternatives on the market already.
> Not only are they not developed by any recognized standards body (meaning they have no long-term sustainable funding model),
I don't see a clear relationship between the sentence and its parenthetical.
I don't know about Matrix' funding model. But I do know Signal, for example, isn't developed by a recognized standards body. What can I infer from that about the sustainability of its long-term funding?
i'd argue that the Matrix.org Foundation is just as 'recognized' as the XMPP Standards Foundation :) In terms of disingenuousness - i'd argue that the definition of disingenuous is accusing us of reinventing the wheel rather than acknowledging that Matrix is an entirely different type of technology to IRC or XMPP. It's more similar to NNTP or Git than either IRC or XMPP. For better or worse, trying to solve existing problems using new approaches is how things evolve. On which note, it's cool to see delta.chat and COI trying a totally different approach too.
> Matrix is an entirely different type of technology to IRC or XMPP. It's more similar to NNTP or Git than either IRC or XMPP.
What does this have to do with anything? It's used for chat and solves all the same problems. You could easily have done all the same things (including building a system that does syncing like matrix nodes) on top of an existing protocol instead of reinventing the wheel.
We seem to have this same dead-end conversation every few months, so indulge me as I try to break the impasse with a story instead.
When I was about 13, I worked on a really fun chat system at school called (wait for it) iNFERNO][ (it was the second attempt at a chat system called iNFERNO). It was pretty awesome, and worked like this: the school ran a large network of 68k-based Macs (LCs, Centrises, even a Quadra) networked to a single Apple Workgroup Server, which ran a Filemaker Pro db instance that the school used for its HR and record keeping. As it happened, the DB server was set up such that anyone could create DBs on it, so the school geek community spun up a few to play with.
If you've ever used Filemaker, you'll know that it's a visual DB - almost creating Forms as the first-class citizen (a bit like Hypercard) and the fact they're backed by tabular data is almost hidden. Even more excitingly, it even had basic networking so you could run network queries between DB instances - including exchanging arbitrary objects (e.g. images, videos etc), if i remember correctly.
So, what we did was to set up a centralised DB (iNFERNO itself) on the server, shared over the network, which stored a bunch of chatroom history and provided a graphical timeline view onto the room when opened up over the network from a Mac (set up to occupy the top 2/3rds of your screen).
Meanwhile, and this is the fun bit: we also wrote a DB instance called your "iNFERNO Key", which you stored on a 3.5" floppy disk and could run as a local DB off the disk on any Mac you happened to walk up to. It coughed up a window on the bottom 1/3rd of the screen, which provided your composer prompt. Everyone had their own key, and customised it to their heart's content with different themes, colour schemes, buttons, bot-functionality etc. And it stored their credentials (and we even tried some basic shared-secret e2e crypto) to make logging in trivial - you just put the disk in the drive and clicked the DB (which would in turn spawn the separate main window to view the shared DB timeline).
When you sent a message (which could be text, or arbitrary filemaker objects like images, videos etc), Filemaker let you stage the table row locally and then insert it into the shared DB instance, so you ended up with a surprisingly fun and usable server/client chat system, with arbitrary scrollback, multidevice support, file-transfer etc... using a batshit crazy architecture which involved schoolkids running around waving 3.5" disks around and forking each other's "keys" and somewhat unnecessarily authoring messages using a local DB instance when the whole thing could have been done on the serverside DB instead.
The whole thing lasted about 3 months, getting increasingly popular until the school went nuts at their Filemaker server being completely overloaded and tragically turned it all off. And whilst I still probably have my Key somewhere, we surely didn't have a backup of the central server itself :(
So, my point is: this was a chat system, with many of the features shared by XMPP, Matrix, Slack, Discord etc today. Could we have built it on top of something else? Sure, we could have built it on IRC instead or Talk or some BBS based thing. But instead we chose to experiment with a totally different architecture, which had its own weird trade-offs and benefits, and learned a bunch of stuff along the way. (I still miss the idea of literally synchronising rich graphical objects over the network - it almost felt like Croquet or one of the virtual world things).
And so, whilst you're totally right that we could have tried to layer Matrix semantics over the top of XMPP or IMAP or Filemaker(!), we similarly wanted to try a totally different architecture; one without stanzas and message passing; one without all the historical baggage of XMPP; one which only does group conversations; etc. So I'd argue this is how stuff evolves, and sometimes it's good to experiment with building on a new base. And if it doesn't work out; hopefully it at least provided a learning experience for all concerned. The same goes for COI layering chat over IMAP; almost as questionable as layering chat over a networked DB ;). So, let's just get on with building stuff and seeing what works and what doesn't. This isn't a case of re-inventing the wheel for the sake of it, but seeing how well things work if you build on an entirely different set of foundations.
+1 and arguably the concurrent clients + multiuser thing is something which XMPP did never get around really. There are at least a dozen or so XEPs which noone is implementing (except for Conversations) which make basic functionality a mess. You download any client and get basic 1on1-chat with no missed messages when both parties are online. Everything else depends on, .... whatever. Even major providers needed some years to get around this (look at Whatsapp). So I think it's really not reinventing the wheel if one tries to implement something anew learning from the pasts mistakes
(if you are affiliated with riot/matrix: keep up the good work!)
well, I'm open to client suggestions (and to public servers, where I can try them out too). I was told by Google (tm) that gajim should be one of the most feature-complete clients on the desktop. tried it with conversations on the phone. group messages go into some archive-window (instead of showing up in the chat window like in every over program), sometimes the archive misses them... pictures are a link to the server, so I have to manually download them all. encryption with multiple clients is a mess (even more so than matrix). av is completely missing. of course this are all relatively minor things from a technical standpoint but they add up and together with the thing that a large amount also depends on the server this makes for a very bad user experience (basically it currently means: run your own server and use conversations)...
> group messages go into some archive-window (instead of showing up in the chat window like in every over program), sometimes the archive misses them...
Not sure what that even means. Worth reporting.
> pictures are a link to the server, so I have to manually download them all.
I think you need to install the image preview plugin.
Public server wise I don't have any recommendations (but i'm sure others have plenty) other than movim. https://movim.eu/#try they have a baked in client (and do a whole bunch of other stuff like micro blogging) but once you have a account, there is no reason you can't use any client you want.
Encryption with omemo is a issue and usability could be improved (I believe people are working on that).
Hey, thanks a lot, those clients look really quite great (and polished). Still, if you look at movim you imho experience the problem of the whole open XMPP-ecosystem. Somehow nobody cares for getting other people to support their features and in the end everything moves closed-source. At this point I think this really is a problem of years of standardizing new features and a lot of projects being sold under the label XMPP being 10 years behind on those features. In theory XMPP today (together with jingle?-based mediation of a jitsi-videobridge) is ready for everything. But in practice I didn't find 2/3 clients you recommended (google+xmpp-foundation site).
I;d like to see a table / chart showing the top 10 or 20 or so xmpp things - and what features are missing from each, along with notes about the language app 1 2 etc is written in, and an estimate in how long it would take coders to add the amount of missing features that others have..
then we could see an estimated total, like for $xx we could get most of these apps all on the same page of features running... if that is the major hurdle - a crowdsourced money or and time to get them all modernized may make the whole ecosystem 2.0 and worthy of use?
Surely it can't be that much that is needed - yet it seems that fixing just one or two is not having the same network effects of having them all speaking the 2.0 feature language.
https://compliance.conversations.im/ not for clients but slowly getting there for the servers. And this is only for a working text-based chat, AV is really of the table. Basically except for conversations, most of the "clients" should be labeled protocol explorers because most of them have serious usability issues with most of the listed XEP...
The worst thing isn't that the standard isn't defined or something, but that a lot of projects are claiming to support XMPP when they only support some core functionality...
> i'd argue that the Matrix.org Foundation is just as 'recognized' as the XMPP Standards Foundation :)
Recognized by whom? The XSF has standardized their core in IETF. Is there something similar for Matrix.org Foundation?
> trying to solve existing problems using new approaches is how things evolve.
What new approaches? You're just recreating federated IM network using different wire format. How on earth is it innovative or new? Using HTTP+JSON instead of TCP+XML is something new? Bridging to other IM networks is something innovative?
> Matrix is an entirely different type of technology to IRC or XMPP. It's more similar to NNTP or Git than either IRC or XMPP.
I've heard this argument many times already, but the truth is that XMPP formally speaking is de juro an "XML routing protocol", but both XMPP and Matrix de facto are being used largely as IM protocols.
> For better or worse, trying to solve existing problems using new approaches is how things evolve.
Since you invented nothing new, you evolve nothing.
The foundations responsible for the protocols look to be pretty similar to me (both non-profit orgs), and would be equally recognized as such by the general public. Obviously XSF contributed XMPP to the IETF after 10 years or so, and perhaps we'll end up contributing Matrix to IETF or W3C or whoever too if they'll have it.
> How on earth is it innovative or new?
> Since you invented nothing new, you evolve nothing.
sigh - I wonder if the XMPP community would spend less time constantly complaining about Matrix if they understood what it was :/
The innovative bit of Matrix is that it's a replicated database of objects (events), similar to Git, but designed for syncing conversation history around in realtime. The events for a given room get replicated over all the participating nodes. There is no central server responsible for coordinating the room; instead all the participating ones do so equally. It's impossible to communicate with someone on a different node without effectively giving them a lazy-loaded HA replica of the room. Architecturally this is about as opposite of MUC (or MIX or FMUC or DMUC or whatever) as I can think of.
It's NOTHING to do with HTTP+JSON versus TCP+XML - Matrix can use whatever transport and encoding floats your boat. For instance, at FOSDEM we showed Matrix running over CoAP+CBOR to try to spell this out: https://fosdem.org/2019/schedule/event/matrix/.
> On its own, Git can only synchronize changes between clones of a repository after the changes are committed, which forces an asynchronous collaboration workflow. A repository may be replicated across several machines, but the working copies on each of these machines are completely independent of one another.
> Memo's goal is to extend Git to allow a single working copy to be replicated across multiple machines. Memo uses conflict-free replicated data types (CRDTs) to record all uncommitted changes for a working copy, allowing changes to be synchronized in real time across multiple replicas as they are actively edited. Memo also maintains an operation-based record of all changes, augmenting Git's commit graph with the fine-grained edit history behind each commit.
They intend to use this in their WIP xray editor - a possible future replacement for the Atom editor. Meno will be used to provide real-time multi-user collaboration for the editor, like Teletype for Atom: https://teletype.atom.io/
I occurred to me, that if your got repo contained chat history, then your edit would then be a chat client, with your chat history version and stored by git.
Been watching Matrix for a long while now, and just now I see this:
> "innovative bit of Matrix is that it's a replicated database of objects (events), similar to Git, but designed for syncing conversation history around in realtime."
and think, this is basically blockchain workings right? IF we had only created a client that worked on iOs only, could store and send chat coins and log votes for each group, this could be presented in press releases as a new blockchain chat and get all kinds of posts and maybe money thrown at it.
kidding aside - I have more faith in Matrix moving forward than any similar projects at the moment, and I don't see that changing soon - so I hope more can be done to polish clients and make sure bridges and other features are well done.
As soon as moderation tools are enhanced I'll begin to use it on several sites.
I'd love to see some bubbles of related issues that need love and have some people put some estimated time and money on each bubble - maybe we could crowdsource some code for some bubbles and money for others or something -
I feel there are many groups of people who want / need different things in order to increase adoption, and I feel for the folks coders / maintainers who are trying to extinguish all the fires at once.
I put a few hundred into bug bounty like thing to get some features added to rocket chat so we could use it.. then found the amount of other bugs left un worked at the time meant it was not feasible for our usage any time soon. Hopefully with the funding and such of Matrix it can evolve on multiple levels at once and become a solid framework with many clients and ux options.
A bridge for COI and delta chat or any others. I'd love a client that bridges with email address and has one click to keybase auth or pgp and such - and puts things into a timeline / fbook feed like view.. not for me, but for others.. group friends, display on phones and web.. seems these things could be close to reality.
Chill dude. No need to get all butthurt. If you don't like Matrix, don't use it. The rest of us will also have make the same choice for ourselves and time will tell whether Matrix has a unifying or dividing effect on chat.
I, for one, have been playing with Matrix and other protocols for awhile and have been impressed not only by the Matrix team's vision, but also by their ability to deliver on it consistently (though sometimes slower than all of us would like). The whole ecosystem has become much more polished over the past year (and the new Riot client is sweet!)
Problem is, "team". It's not really decentralized at this moment, just software from the singular vendor. Of course, it moves faster than a really distributed protocol (in all senses of this word).
Let's assume this hypothetical client doesn't exist (I'm not actually sure if it does or not): are you suggesting it's better to start over and create an entirely new chat protocol instead of just writing a client that has the features you want? These features do all exist within existing open protocols, and clients do exist that implement some of these features (and probably all, but I'm not sure). So I don't see any reason to start over.
I've been using [Wire](https://wire.com/en/) very happily for some time now. Great interface, top quality audio/video, good chat, persistent, very secure, open source. I'm surprised it hasn't received more mentions.
promised for 6 years... and remember this thing is commercial, not OS...:
"Nonsense comments from clueless commentors. It's not our problem if you didn't update on feb14. Also, it's our app and we do whatever we want, please uninstall Xabber and use something else. You had no voice or rights here besides those that have been generously given by us. Now they are revoked."
Jabber is dead. Apparently it was either bad enough or complex enough to prevent creating good clients and servers. Matrix is another chance to kill Whatsapp and Telegram and replace proprietary software used by billions with free software. I'm not sure if it'll be successful, I think that implementations are not there yet and momentum might be lost when they will be there. But Jabber won't recover and we should try, because giving up is not acceptable.
To counterbalance this a bit, I'm using it heavily with a large group of (partially non-technical) friends and friends-of-friends, along with participating in public rooms over federation and it's been pretty smooth. Things were certainly rougher a year ago but they have improved a great deal and are continuing to improve at a good pace.
Speaking in my personal capacity but as somebody's who's a freenode oper:
The ridiculous crashing every few minutes thing of the IRC bridge has been fixed enough I haven't noticed a problem for quite a while.
I have a looooot of disagreements with matrix, which I have discussed and will discuss with them, but honestly they're not doing it completely wrong anymore.
> I have a looooot of disagreements with matrix, which I have discussed and will discuss with them, but honestly they're not doing it completely wrong anymore.
\o/ :) The other stuff is very much on our radar, fwiw (and in an ideal world i hope we'll finally be able to implement a services or TS6-based bridge). We've been a bit distracted over the last few months with an XMPP bridge (matrix-bifrost) but will be back on IRC bridging shortly.
This is completely the opposite of my experience. I'm in probably a dozen Freenode channels via bridge and another 5-6 Mozilla IRC channels, and haven't had a problem in as long as I can remember.
Bridges usually suck, even for IRC. Split brain, someone needs to monitor it constantly, you are not sure if Matrix clients have received it and vice versa.
But any bridge will be restricted to the lowest common denominator, often you get into weird corner cases where the two ends of the bridge work subtly different.
Very true, but have you tried just reaching out to someone via email on Matrix in practice? I tried with Riot and am not able to do that. Maybe just a bug, but if you want the real Matrix experience, you need a Matrix account. Don't get me wrong, I do love Matrix, but I also love XMPP - both not spectacular successes so far. AFAIK, the French Matrix version is even locked down and does not federate across the network.
You can reach out to folks by email in Riot by hitting "start chat" and entering an email address. I'm not sure how much easier we can make that ;)
We should run a general purpose email bridge though, and in fact there's a massive public sector outfit asking for such a thing currently so it'll probably happen sooner than later.
In terms of Matrix not being a spectacular success - well, that's a matter of perspective. The intention for France is absolutely for it to federate with the public network (once there's are sufficient border gateways in place). And France is far from the only high profile Matrix deployment out there, it's just one that we can speak about. But eitherway, COI looks cool, and I hope it helps OX sell more mail servers :)
Looking at the documentation this looks to be at a very early stage, but after reading it I think it's mostly on the right track to become a very interesting platform...
It's decentralised by design, has support for most things people expect (read notifications, group chats, polls, images, video, location, contact), almost every feature is usable by someone with a regular e-mail client and integrations should be very easy to develop (any e-mail library is good for developing a COI bot). Looking forward to a client and server to play around with... maybe I'll make my own as a toy!
The aim seems to be to create a system that can replace centralized messenger solutions by allowing use of existing IMAP servers or hosting one yourself. I am not sure if this is correct but the article states that they are already supplying the software for three quarters of all IMAP servers worldwide and seem to be interested in connecting with Google to bring this to fruition.
Pretty interesting read (although only German) but it seems the marketing on this topic may have just started.
Looks like native COI compatibility on the server makes life easier for a client, and adds advanced features as opposed to restricting some servers from the party:
>"A COI client should check for COI support from the server first. it will CAPABILITY after having logged into the IMAP service. If the returned capabilities list contains the "COI" capability, then the client does not need to filter messages on the client side. It will also have more advanced options at its disposal, see the server side specifications for details."
The requirements are just an IMAP (or JMAP) server that conforms to the IMAP RFC:
>"The IMAP server may or may not be COI-compatible. The IMAP service MUST be compatible with IMAP v4rev1 as defined in RFC 3501, which is the case for almost any IMAP service in production."
An SMTP service for sending messages. Sending services are also called Mail Transfer Agents (MTAs)."
We have about 7bn total unique active (3 months) accounts in the world, 50-60% is Gmail, Outlook.com, Yahoo and a few more, 40-50% are real IMAP servers, of which 76% use Dovecot. Very large deployments include companies like Comcast, Deutsche Telekom, 1&1, LGI (Ziggo, Virgin Media, UPC) with 10m+ user deployments, and yes, a veeery long tail of millions of small servers and anything in between. But COI will also play fine with GMail and friends.
Delta Chat is here, now, and has been for a while. It works on Android, Linux, MacOS and is being developed for iOS (limited function beta is available). There is no Windows client yet as far as I know. Delta chat has a simple and concise website which quickly comes to the point and does not try to upsell anything.
This... has a 'modern' website and not much more. The site starts with a page-filling image of a happy woman in an urban environment, she's clearly exited over something the lighted slab in her hand presents her. The thing sorely missing from the site is the download link and the link to the source repository where all those dreams are made true. It does contain one interesting piece of information though:
> COI for Developers
COI is the only ecosystem that provides openness, a strong messaging basis AND overcomes the network effect.
With COI you don’t need to develop, host and maintain your own communication servers. Thanks to Delta Chat Core, as a client developer you do not even need to deal with IMAP directly.
Thanks to Delta Chat core... in other words, this is based on Delta Chat. According to one of the project founders they are cooperating with Delta Chat to ensure interoperability [1].
We work with the great Delta Chat community and contribute to it. COI will enhance IMAP and the email formats for clients like DC to be able to provide more robust real-time chat and channel functionality and to make implementation of chat clients on email easier.
Primarily I guess this is an open specification, whereas deltachat is a closed app? sounds like the same underlying principle to me though. Apparently in the future there would be optional extensions on IMAP servers to make things more efficient.
I really like the idea, but I prefer dedicated self-hosted services like Matrix.
You can run a suspicious Matrix instance, but I think privacy falls away as you talk to any Gmail user via CoI.
"With XMPP and Matrix.org -based services you would still need to convince everyone to join your new network. Easy in theory, very complex in practice!"
I don't find that so problematic. People just needs to see a bit the inner workings to feel a bit comfortable and give it a chance.
You can also run your own mail servers, even non-technical people could do that with for example https://thehelm.com/. Haven't tried it out myself, though.
So due to HN cross-pollination, I'm now using Delta.Chat on my android which works a treat.
Hopefully once COI has a client, I can try that one out as well; if its android client supports multiple email accounts per install (unlike Delta.chat at the moment), it will almost certainly make its way into normal app rotation and evangelizing BBM/WA/Messages users to a more private, secure instant message client.
At the same time, I have to worry: SMTP is forever, but I'm not so sure that IMAP is forever.
Email is increasingly becoming a fragmented oligopoly, and the major players are promoting their own alternative protocols. IMAP is becoming a very optional legacy feature.
If the success of this project required additional integrations with EAS and Google's mail APIs, would you do the work or reject the idea on principle?
> SMTP is forever, but I'm not so sure that IMAP is forever
Indeed. I was surprised to see that Tutanota, a German privacy-oriented e-mail provider, only provides access via their web front-end. They don't support IMAP because according to them, they couldn't provide end-to-end encryption in combination with full-text search.
This is great and all, but you're relying on IMAP, an inherently mobile-unfriendly protocol. They should have worked with Fastmail to get this into JMAP.
The specs will soon be available on GitHub soon. If the server supports JMAP, there are no principle obstacles to also support COI over JMAP. But as said, currently the market share of JMAP is neglible.
i assume for this to work remotely ok the imap idle is a must.
also how is it supposed to work on the receiving site ?
There has to be some "Folder" where those messages are stored and the imap idle should listen to that folder or else my inbox will be huge mess and mess with my regular mail client.
also was there any tests how long does it take for delivery, because the message could hop around and could be scanned, added addotional headers or mess with those there.
If it's basically two mails clients sending emails quickly to eachother, is there anything there that will make sure the user doesn't fill his inbox with tons of "hi","how are you" emails?
Does it delete the messages that were read, or something? If it's really going to put all the messages,is it using some kind of dedicated folder?
Chat messages will land in a separate folder for the servers that support COI. For the others they will be collated into normal mail messages with a small delay, but yes, it may end up being many.
I think this could be a great solution, but consider going after Slack rather than Whatsapp to being with. Unusually I think enterprise customers will make better early adopters than consumers, as they don't care about network effects and their technological inertia actually aligns with your selling point - that it is a thin protocol on top of email.
Once people use this for their professional email address they may start to using it for their personal one too.
No, that’s right, they don’t care about network effects; enterprise customers care about a SLA above everything else. An open protocol cannot provide a SLA.
I have seen other users bring this up, so I think it should be stated in its own parent comment: most of the public uses yahoo or gmail to host their email, so chatting with them is all tracked by yahoo or google. That is something that needs to be addressed.
I like this idea. I could build a client and compete in the messaging space with this tomorrow. As a developer having come of age in a world ruled by FB Messenger, iMessage, Whatsapp etc, this is a whole new freedom and opportunity.
I'm aware that these battles have all been fought in the MSN/ICQ/AIM era too, but it looks like we're going to have to fight them again.
It is very difficult to build a decent email client with IMAP. So initially thought it is a no-sense. But after reading the while proposition seem very intriguing. Although it is based on email adoption will be very difficult.
Once you try to integrate with different systems, you'll find one of the major difficulties of IMAP: nobody really cares. Every player is different. Even gmail has its own issues, like creating "folders" for each tag and duplicates the same message to each folder, so your little simple client now has custom de-duplication logic.
Working with IMAP at a large company was my first "meh, how hard could it be" hubris moment. Their implementation must've had a decade of work put into it. Seemed bizarrely complex to me, so I started a side project of building my own. My little hero developer saves the day fantasy. Could've made your same HN comment early on when my PoC integrated with its first service. My side project never advanced beyond weekendware after I started seeing why it was hard.
Eventually, the guy that worked on it all that time quit over some political drama. They asked me if I wanted to maintain the project. I tried and basically quit over it.
Also, IMAP itself sucks. Did you know the spec doesn't require emails to have a message id? Also, no servers have to return your commands in order, and they don't have ids. So you'll code shit like "if the response looks like this, it must've matched up with this one command." I could go on and on about the nickel and diming that will happen to an IMAP client that you thought would once be simple.
> Also, no servers have to return your commands in order, and they don't have ids. So you'll code shit like "if the response looks like this, it must've matched up with this one command."
I'm sure you know, but in case someone reads it and doesn't know it this is perfectly as intended: IMAP isn't a request-response protocol, it's more of a pub/sub protocol.
I'm implementing mobile client too :)
To sync all mailboxes, not just Inbox need a lot of commands (request/response). Which add a lot of latency which is not very user-friendly on mobile.
For example currently, I' exploring JMAP and for the same amount of functionally, the API calls a just 1/10th of what needed with IMAP.
It's not unusual for Exchange servers to have IMAP support enabled. Many organizations have bought into Microsoft's subscription model, use O365 for email instead of running Exchange servers themselves, and typically won't disable IMAP access.
POP-only mailboxes are pretty rare these days. A lot of users still use POP, but I guess mostly because they configured the mail client a decade or two ago and are not touching it again.
I like the spirit of this idea. I'm glad to see existing technologies being leveraged. However, it still seems quite a dog's breakfast of parts and pieces that don't all seem to be necessary.
I'd like to see this split out into several parts, which could then be used together if necessary.
The basic idea -- how to format and transfer short-form (chat-style) messages using existing email standards -- is brilliant. This seems similar to how AMP describes a limited set of HTML, and would be worth defining carefully.
The management of distributed contacts and group lists seems to be a different problem entirely. Personally, I don't need this: I'm fine with using my existing address/contact lists of trusted friends, along with time-tested tools like mailing lists.
The idea of editing/deleting messages seems superfluous and complicated. I've never knowingly used a chat system that provided this feature, nor have I ever wished I had it.
Maybe I missed something, but I didn't see anything about how SMTP would be used. It seems that by the time a short message is wrapped up into its attachments and headers, then sent to to the appropriate SMTP server (along with login & negotiation), I've probably sent over 20KB. That's a lot of data for one tiny message.
I would love to be able to use COI without a special client at all, just my current email client. However, the spec seems to require certain headers that would generally be difficult to configure in most email clients.
I'm concerned that the spec does not include any discussion of error handling. There are many things that can go wrong, especially with an asynchronous protocol like SMTP/IMAP. How are those problems going to be handled?
I mean metaphorically sure, but in this case it's literally using the exact same system; it's just the clients are changed to look like a chat app instead of an email app. This is more like if two people were sending telegraphs over actual telegraph lines, with morse code and everything, then at the end they type it up into a Word document and say "It's email!"
Wouldn't it be possible and transparent to the protocol to use PGP for end-to-end message encryption? If the recipient has a public key, the client can automatically use that and the recipient's client could automatically decrypt it.
I wonder how COI will perform battery and latency wise. I mean, if an email is delivered within 30 seconds to the recipient, that is considered fast, but if instant messengers take that long you wonder what is broken.
That's a funny take, considering my experience. I know some email servers are slow, of course, but they don't have to be. When I worked on Gmail, the majority of Gmail-to-Gmail messages were delivered in under a second, and not because they have a special internal protocol; even such internal messages are sent over SMTP. But Gchat was never that fast. 1 second was considered fast delivery by Gchat but it was long tail slow for Gmail.
>90% of mails are delivered really in a couple of seconds, but yeah, turnaround could be quicker. That's why in the longer run we hope to skip the store-and-forward mechanism and instead deliver mails directly by IMAP server to IMAP server.
I doubt most email servers actually take anywhere close to 30s to process an email. I suspect most of the delay is the sending server queueing and email client polling intervals.
That's like complaining that your web browser gets full access to your browsing history, or that your IM client gets full access to your plain text conversations, even when using E2E encryption no less!
No, it's a client that uses your email account for purposes involving reading and sending email. Exactly like Thunderbird, or Evolution Mail, or Outlook, or Android Mail, or iOS Mail or countless other clients that have full access to your email client by design, and without issue.
People generally take issue with handing over access to their email account to third party services. People don't generally take issue with a client under your control accessing an account under your control.
E-Mail got extended as much as it could and COI doesn't change anything about it.
If you want something like COI you can use deltachat https://github.com/deltachat/deltachat-android
I couldn't find any specification about anything (encryption, contacts, group-chats,...) and the worst part is that it doesn't even solve the arbitrary problems it poses itself.
privacy? Fat chance when you use E-mail.
"big bad providers" (yes that is the summary of all the other "points") you still have those too
And the solutions that are there and work like matrix or xmpp are no solution, because of user adaption? Hello... Users have to adapt your "protocol" as well and I don't see any added value... It's just an idea for chat over email which I can do right now with the right Client.
Ps matrix AND xmpp both support bridges for other protocols.
Pps they miss the point of what is needed most right now/what the problem is.
That doesn't mean the connection between all servers is encrypted (smtp can be routed through multiple servers) nor does it mean that google or whatever server the data get stored on/routed through can't read your communication.
The whole thing just sets a low bar because it can't implement all the things other messengers already support (for instance not disclosing who you are talking to)
I'd like to try this or the similar delta.chat, but have thousands of confidential/sensitive e-mails on my account and don't know what these clients will do to/with them. It's also not an ideal situation from a security perspective (a lot of damage could be done if the clients are exploitable).
I understand that from a protocol perspective this is more open, so your data is not going into another silo like Signal etc.
From a consumer standpoint, isn't it still true that you still need to get people to use the new client(s)? (It's not like gmail/hotmail/yahoo/etc. email service providers automatically have a COI client on top of their email client) In the end, you still have the network problem, of getting people to switch over?
That would make statements like this a little bit deceiving: "You can also reach everyone, there are more than double active email users than WhatsApp users, for example."
Yes, all email users (who use services that support IMAP) are technically automatically "users" of this, but before they start using your client, you still can't chat with them over COI. Or is there fall back to email?
Hey everyone. I don't want to start a debate or anything, I'm just looking for a privacy concerned chat service that I can easily teach my fiance to sign up for and use. She gave up on Pidgin, and I would appreciate any pros/cons for your favorite service of this kind. Thank you in advance!
We use Wire. In some respects it's more useful than Signal, because it works on mobile, desktop and web. I think the desktop clients are Electron based, so I use the mobile and the web apps. You don't necessarily need a phone number, it can also use an e-mail address, so it works just fine on the desktop. It also has voice chat and conference. Oh and a good part of it is open source, although it's uses a centralized server.
I second the Wire recommendation because you can sign up with an email address, and conversations sync across devices. Its feature set is also good (not as good as Telegram) and improving. I wouldn’t recommend Signal as it is still poor in features, speed and stability.
With Signal you cannot carry your chats to a new device (this is prohibited by design in the app on iOS)!
If you want to experiment, try both Wire and Signal at the same time, and you’d observe the differences quickly.
I personally love Threema [1], which is available for iOS, Android, Microsoft and Web. It uses a random ID to identify users, uses End-to-end encryption, you can make voice calls, send messages, audio messages, GIF’s etc.. Oh, and it has a dark theme!
Please note that Threema is not open source. If you want that go for Signal instead.
Interesting comment. How frequently do you think this one vcard-related issue you linked to is a problem for Signal users trying to share contact information with each other?
And could you give an example story illustrating how two users would run into this issue in practice?
I'd recommend quicksy if she's really clueless. It's an xmpp client that uses the users phone number as identifier and does autodiscovery with the users address book to check if any of their contacts are registered too. That functionality aside it's still a fully functional xmpp client, so you can talk to her with conversations, gajim, dino etc if you want.
it's like the only possibly important thing about this and nobody has mentioned that the COI web front runs into nothing when you follow URL toward an RFC:
The biggest problem with IMAP from my point of view is spam, that is why there is only a couple of email providers in the world, although still decentralized, they trust each other more than any newcomers, which is kind of preventing this initiative.
We thought of that a lot. But like our other baby, http://id4me.org - where we chose use use DNS and OpenID and mesh it into a federated identity protocol - email infrastructure is simply there, it's federated, robust, run by millions of servers, large and small, tested, filtered, pounded, drowned, abused you-name-it. That buys some extra overhead.
"Last but not least, every COI client app can offer a variety of end to end encryption options to keep your communication absolutely private."
The fact I had to read quite so far down the page to find this snippet emphasises to me how unimportant the developers of this protocol see E2E encryption to be :( And leaving it to the client seems like a terrible idea, especially if there are options, as then surely it will depend on what E2E encryption options your and your recipient's clients supports?
I don’t think that means the developers don’t consider it important; it means that the people that want E2E aren’t necessarily in their primary audience.
The prime target of this protocol isn’t power users and developers; it’s the overwhelming majority of users that stick to siloed messaging because it’s convenient, and everyone has it. First and foremost, they need to be shown that this is more convenient, and has a larger and more accessible network.
It can’t be default because the protocol is degradable by design. I would agree that they could advocate a method to harmonize all COI-aware IMAP servers. But it’s hard to get E2E encryption in a heterogeneous, legacy environment.
The draft spec available in the confluence wiki says the following:
>A COI-compliant client MAY support the Autocrypt standard to ease end to end encryption scenarios.
>TODO: Consider using more secure lookup mechanisms for encryption keys. Also check for existing encryption keys before auto-generating a new encryption key set.
IMO its comparing itself to Whatsapp and other messaging apps that do have end to end encryption. Thats why the current email trust model is not as important.
I agree. Fascinating, but its not exactly 'Signal' (end-to-end encryption with perfect forward secrecy) over IMAP. That would be something I'd jump on, fwiw.
To tout it as an alternative, or "breaking free" of WhatsApp, is to discredit one of the pivotal reasons that people use WhatsApp, which is E2E encryption.
The pivotal reason people use WhatsApp is because it can be used without paying (with money that is), and because up to 90% of people in any country use it¹.
That's it. Facebook could remove E2E today, and lose only a fraction of its WhatsApp users. Of course many little annoyances might mean that a competitor may step up, but WhatsApp is incumbent, and this is no longer about features or technology. It's all about the fact that your friends, family, and colleagues are using it. It can safely ride the network effect for years.
I welcome any protocol that makes chat something that is available for everyone with a computing device again — not just on approved smartphone operating systems.
Not to forget that WhatsApp started out without E2E and didn't implement E2E for quite a while.
WhatsApp simply won the race by being there early on and getting the critical mass. From there on out people are locked in, because everyone else is there.
I would add: WhatsApp was there, free and very well done.
Even in countries that had free domestic text messages (where the “free” factor was much less important), WhatsApp nailed group messaging better than anything else available at the time. Also, by using phone numbers they skipped the entire “create account / invite friends” stage, making for a much smoother bootstrap.
Certainly it varies a lot. But back in 2017 they had 1.5 billion monthly active users (20% of the world population). The US is probably an outlier here (I hear SMS is still popular in the US whereas I literally can't remember the last time I sent an SMS, it must be over 5 years ago)
Same in UK, I installed it as almost all my (relatively non-technical) friends and family were using it. It's much more commonly used than Facebook, or email, for direct to groups communications.
You can in theory. But you have to choose: Either your data is encrypted or it's searchable. So in practice you prefer searchability.
Someone will probably say something about homomorphic encryption. Not deployed AFAIK, and deploying will be difficult if you want to ensure that you can search your encrypted mail but others cannot.
Oh, sure. Just download all of the messages. The problem with that is that you run into users like me, who have 25 years of archived email, a mobile phone, and who travel abroad.
Are you suggesting that if I want to search on a particular device, then that device should hold my backup storage? If so, then I disagree and have chosen to store my backups on a device that's behind two locked doors, hard to reach, well protected from being accidentally disturbed, and which doesn't run unrelated software.
Okay, suppose that I were to back up all of my mail to each client device; three at the moment, or four if you count my old phone, which is still running because I haven't gotten around to wiping it. Then I'd have five backup sites instead of one. But four of those are backed up to the fifth, so the size of the mail backup would be quintupled, the activity on the backup device would increase and with it the number of possible occasions for a failure, and the age of my oldest backup would be automatically decreased to compensate.
I work for a large consumer facing mail provider. Imap is easily 60% of our cost because it's a very inefficient protocol. Also, we can't put ads in 3rd party imap clients. So we make zero revenue from them.
COI will exacerbate the problem. I wonder how we will solve this problem.
Are Google, Apple, Microsoft, and other large email providers likely to implement a protocol like this into their own servers? It seems like a major conflict of interest, and without them it is a massive portion of the market that will never see this service work.
If I interpret the homepage correctly clients/servers without extra COI enhancements will just receive the messages as emails. Not ideal, but in any client with threading (gmail, apple), it would work decently i guess.
According to the spec (https://confluence-public.open-xchange.com/display/CoiW/COI+...) clients are expected to ask the IMAP server using a CAPABILITY command whether it supports COI extensions. Whether or not the server does support COI, messages are indeed standard mails with an extra Chat-Version: header and the requirement that Message-Ids start with "coi$".
So an existing server need not do anything to support COI clients. But a malicious server could easily decide to reject them, or to filter out COI-specific stuff from messages. I don't think it's likely, but it sometimes big corporations and governments get paranoid about weird things ("network load!!1!") and take technical measures to block them.
I wouldn't be surprised if COI encounters issues with automated abuse detection and spam filters in those servers. When you're in an active conversation, the client is rapidly sending 1-line emails with strange headers.
I've browsed through the spec and it looks like servers likely won't need to change anything. Most of the interaction between clients happens via mail headers.
They have a description of the client but I could not find a description of the optional server-side enhancements (besides that it should be advertised via CAPABILITY).
Chat over IMAP seem a bit misleading title.
Their proposition is to add additional info in MIME headers. Or later some of them like Message-Id with chat info.
So seem more related to MIME and SMTP.
A client app can get those headers with different protocols too. MAPI, EWS, JMAP, etc.
Chat over SMTP/IMAP/MIME just doesn't sound right, or does it? We also thought of Chat over Email, or Chat over Mail, But Chat over IMAP gave us the COI (fish), which we love. Decision made :)
"COI uses an email address and any IMAP server as its infrastructure. This means it can already connect 3.8 billion users - anyone with an email address"
Email <> IMAP even though most email providers do provide access to IMAP, this is greatly oversimplified.
Seems to me that XMPP already covers all these use cases of COI? Both XMPP and email use essentially the same addressing format but XMPP uses XML formatted messages whereas email uses HTML formatted messages (it's worth noting that XMPP used to support HTML messages).
So you could "send an email" in XMPP just by composing a message and sending it to what amounts to an email address (technically the bare JID). The XMPP client would just need to close the chat window after sending the message to make it seem like typical email experience.
XMPP also has the advantage that it was designed with the IM and multi-user chat use cases in mind (e.g. chat state notifications, message corrections, moderated/members-only chatrooms, etc.) but also can support email use cases.
I think it's easier just to set up your XMPP server to handle IMAP authentication.
Anyone else getting a Cisco - warning about malware.opendns.com on the Wiki? wiki.coi-dev.org. It's blocked to me at work on this url. wiki.coi-dev.org&server=lon16&prefs=&tagging=&nref
This would be awesome to make an embeddable, html form, so essentially any website you want to share links with your friends from, can be done on the website itself coming from your email address.
And yes, we want to open the chat ecosystem to many many cool clients made by you all for every taste and use case. Beyond the obvious WhatsAppySnappyChattySlacky varieties, if you think it through, you can even envision Twitter and Facebook-like clients. Sounds crazy but we cannot imagine why not. Plus normal Email clients can be seriously pimped.
Fastmail generates a password for applications that you give access, so you never actually share your real password. I imagine they could use a solution like this.
Current thinking is that the IMAP servers connect directly, but via SMPT for message transfer, but without the store-and-forward, once they are trusted via a token exchange. May be blocked though in server deployments (ports).
Very bad choice of design for the website. I'm supposed to be interested in this, but this website looks like they're trying to sell me some multi-million contract.
Too much white space. They could have condensed it quite a bit. And who needs a hero image that big anyway? That was "modern" when people just started using the word "modern". Like using the word "modern" now, it's pretty dated.
All the same email is pretty siloed, for personal addresses. Running your own server is a hassle for anyone,and the most popular provider stopped being "not evil" some time since 2004 when I got my address there.
Running your own server is not a hassle for "anyone". I wish people would stop stating this like it's a fact. Having maintained communal SMTP/IMAP servers for the last 20+ years I can tell you that sometimes a year goes by where I don't have to touch it at all.
it works over regular smtp and imap, via your usual relays and email servers, etc. If you can send and receive email, then I guess it should work fine.
If you can send and receive email, then I guess you have been bitten at least once by its store-and-forward, best-effort nature. In other words, no guarantees on message delivery time (two days delayed? Working as designed!), message ordering, or even whether it's delivered at all. I literally know of no worse medium for chat than email, even considering raw ethernet frames and pigeon post.
"It is sort of available everywhere" is email's only selling point - that's not enough to justify "let's build everything on top of it".
IMAP is basically just a way for attackers to test credentials. You should be very very careful about enabling IMAP. Hopefully you (or your provider) are watching the logs and can respond to abnormal login attempts.
I think they missed the the bit where Matrix is called Matrix because it bridges (matrixes) the existing networks (Slack, IRC, Telegram, Discord, XMPP, etc) in, rather than needing to convince everyone to join. But no matter, we'll just provide a COI bridge if this takes off :)