When things like this happen, I start thinking how much we got some things wrong, especially things like chat.
Here I sit, with my colleagues, connected to the same network, with the same application open, but we cannot write to each other, because some backend server in the US have a problem.
How in the world could this not have been fixed yet? Our internet backbone addiction is seriously crippling us when things go sour. Not thinking specifically about Slack, but other more mission-critical tools that just... stops working because they cannot ask the time from their backend server.
We seriously need P2P systems (without blockchains please) that can be used by normal applications to solve issues like this. Offline and local network communication should be obligatory today.
There are P2P systems already, plus you can run an IRC server locally. But you don't.
Using a service like slack has downsides, as you are seeing now, but running your own service or using P2P has other downsides that we have deemed are even worse.
Maintenance, lack of client support, network access requirements, lack of features, etc.
Obviously right now the downsides of a central service is front and center, but we can't ignore the downsides of the alternatives.
and yet here we are, decades later, and email, www, irc, ftp, DNS, etc, etc. all have a wide choice of clients and servers, of all shades of FOSS licenses, commercial options, hosted options, etc, etc.
IMHO a huge problem with the Internet since the late 90s is that business models have replaced protocol RFCs.
I agree with you, and I also mentioned it in another comment. We don't have to chose P2P or hosted service, when we can have both. See https://news.ycombinator.com/item?id=16109202
As my full-time work is working on P2P systems, I would love to hear more about what you think gets even worse with P2P systems compare to hosted services.
Lack of client support, features and so on is more a sign of lack of adoption, rather than something fundamentally worse with P2P than hosted services.
Biggest issue is connectivity issues. Corporate firewalls don't allow incoming connections, and limit outgoing connections to specific ports (in most cases).
Maintaining history is also tricky in a P2P system. Can I search and view all chat history from when I am not there? A hybrid system could, but then you would still be limited by the host being online and accessible.
Also just the amount of moving parts. With P2P, you can get strange issues where some people can talk to some people and not others. It can be hard to track down where those connectivity issues are.
My company moved from self-hosted IRC to slack a few years ago. No one had time to maintain the system (backups, updates, troubleshooting, etc), and it was cheaper and easier to pay slack than to hire someone to maintain an IRC server, or to pull someone from their actual job to maintain it.
> Biggest issue is connectivity issues. Corporate firewalls don't allow incoming connections, and limit outgoing connections to specific ports (in most cases).
The STUN protocol solve most of this issue[2], webRTC is working fine in the majority of enterprise network.
> Maintaining history is also tricky in a P2P system.
It's perfectly possible to build a distributed database, actually that's exactly what blockchains are (OK, I know , we said no blockchain but I merely use it as an example ;)).
> My company moved from self-hosted IRC a few years ago. No one had time to maintain the system (backups, updates, troubleshooting, etc), and it was cheaper and easier to pay slack than to hire someone to maintain an IRC server, or to pull someone from their actual job to maintain it.
This isn't p2p, p2p shouldn't be confused with self hosted. Nobody has time to set up and maintain a server of any kind, but everybody was seeding torrents in the late 2000 and nobody complained about it being too much work.
[2] while the TURN protocol gives you the “hybrid system” for free (development-wise, you still need a hosting plan in that case, but it can be mutualized between all your applicative needs).
I'm all for P2P solutions and a more distributed Internet, but I'd like to add a few items to that list.
- Security in a P2P system is harder to get right. The system is distributed and each member has to trust more parties. This significantly increases the attack surface compared with a client that just needs to trust a single server.
- Not everyone is connected all the time, so network partitions happen constantly. It's hard to maintain a consistent state such as a chat log, not to mention anything more real-timey like the state of some real-time game.
- STUN and other NAT traversal techniques don't always work. NAT is horrible, creating connectivity between NATed clients is always a dice roll.
- Bandwidth - A dedicated server can have a lot of upstream bandwidth (to the clients). In homes, it typical to find (in some countries at least) connections that are heavily biased towards the downstream. So a 100Mbps connection (downstream) might have a 2Mbps connection upstream. Now say you want to have an 8 people video conference call, and all of your users have a 2Mbit uplink and need to send 7 copies of their video stream... What if it's a 20 person video call? What if half of them are on mobile?
- Bootstrapping - every P2P system still needs to have dedicated servers for bootstrapping. There are no pure P2P systems AFAIK.
- Client versioning and patching - a lot harder in a P2P system. With a central server - you just update the server and everything works. With a network of clients, things are trickier and you run the risk of partitioning the P2P network between different client versions.
You're right in all points (almost[1]), but those are mostly technical issues that can be solved with some work[2], except for two points :
- the bandwidth problem is real, but there is no more technological limitation[3], it's merely a commercial problem: client have no use for huge uplinks, then providers don't sell them. It's a bit of a chicken & egg problem.
- the monetization problem is the root of all sins in that case …
[1] on the client versioning & patching side, you underestimate the complexity of distributing a web client, with CDN, browser caches and clients who never reload their page, you can also end up with a lot of client lagging behind in updates.
[2] and centralized system also have their own issues (scaling anyone ?)
[3] ADSL was asymmetric because the asymmetry widened the reception bandwidth, which would otherwise be too low. With optical fibers, you don't have such limitations.
Plus one on this -- it would be nearly impossible to make a P2P chat system work on our corporate network since we have layers of isolation between users (not to mention about a dozen small branch offices, each with their own independent office, so even if I can Slack the coworker next to me with P2P, I couldn't reach the coworker in Germany that I really need Slack for). So we either need a central chat server on our network, or we use a cloud service like Slack.
We used to run Microsoft Communicator (OCS) and availability of that service was far worse than what we've seen with Slack.
> Biggest issue is connectivity issues. Corporate firewalls don't allow incoming connections, and limit outgoing connections to specific ports (in most cases).
NAT's are the real pain for this sorta stuff and why people use stuff like skype, slack, discord, etc. When the majority of folks have IPv6 connectivity a lot of things should become much easier (at least for home users and people not behind a frustrating firewall they don't control)
I'm surprised maintaining a small IRC server was that much of a time sink. User problems is understandably another story.
You would have to run some servers on the inside and have a specific tunnel to a small set of servers running on the outside. Clients could see either. I don't see how this is an issue.
To me, the biggest issue with something like IRC is that any messaging system that only supports client-side search of message history feels anemic in 2017. It needs to be seamless to use the system across multiple devices some of which will not be always online, which requires server-side history and search. Maybe you could cobble something together to achieve all of that using IRC, but even if you could it wouldn't be anywhere near as seamless as something like Slack.
The funny thing is that even with p2p systems many people use them only in a centralized mode. At my last company we kept our code in a private github repo. When our internet connection went down two devs found that they “couldn’t collaborate because they couldn’t get to github.” It didn’t occur to them to simply push their code back and forth between their machines.
Fair enough. What about having both? Slack could by default communicate via their backend, but if the client finds peers on the local network, prefer those. When the connection to the backend dies, only use the local connections.
While the redundancy would be nice, I can't see it being worth the effort/money for anything but the most critical of systems (which should probably be self-hosted in their entirety to begin with). If the internet (or some major swath thereof) is down, that's going to cripple a lot of businesses these days regardless.
I just don't see it being worth the extra effort of self-hosting a redundant copy of the system, and then keeping that system up to date with the latest software for security patches / features / bugfixes. Especially when the cloud version has n x 9's of uptime.
Just curious - does Slack offer SLAs? To many businesses, especially remote businesses, Slack is becoming more and more integral to day-to-day operations.
I don't get why people think an sla is any benefit whatsoever hosting or software services are generally so cheap, any refunds are completely superfluous compared to the business and productivity loss.
I have never seen an SLA pay out anything more than a small multiple of the hosting cost which still of course doesn't amount to a hill of beans for essentially anyone. Have you?
>If we fall short of our 99.99% uptime guarantee, we’ll refund customers on the Plus plan 100 times the amount your workspace paid during the period Slack was down.
Maybe an emergency mode in the Slack application. When users cannot reach the Slack servers (when they're down or bad network connectivity) the application exposes a P2P connection and looks for other P2P instances.
It would still need a centralized tracker (like a torrent tracker) to help P2P connections find each other through the internet.
You're just pushing the target aside a bit, comparing an external mothership to running an internal mothership.
Why have a mothership at all? Why couldn't software clients simply talk to each other? We already have hash-based addressing systems that can find known peers anywhere in the world (e.g. Tor).
Because in many corporate networks, it's easier to talk to a server half-way across the world than to talk to a laptop computer on the desk next to yours.
Exactly. And by rare, we're talking 99.99% uptime. Slack is down for me too, and I'm not exactly excited. But this is a mild inconvenience compared to what I'd have to deal with should I have my own self-hosted alternative.
Back in the early 90s I recall using a program that allowed two or more people to chat at the same time, without a server. It was a true peer-to-peer communication program. The current paradigm of Internet usage has blinded us to what is possible.
Didn't AIM have that as an option? You had to "direct connect" to send pictures. Of course, you had to connect to their servers first and the parties had to agree to direct connect.
I'm also pretty sure all of us wrote this in CS 101. I sure did.
Honestly I've heard this before and maybe I'm overly bitter, but this just sounds like an IT department that doesn't want to do their jobs. You're there to manage those internal systems. If they're going off the rails that often, it sounds more to me like you're just shit at your job.
We manage over 900 virtual systems on many, many hosts and file systems and have next to no downtime because we work really hard to make that the case.
SIP for instance is perfectly happy to do ip<->ip calls (well ignoring NAT issues - one day we will all have ipv6 addresses....)
This really is a solved problem, people for various reasons have decided to rely on external services they have no control over. I don't think we need to throw yet another protocol at the problem, instead we should focus on making what we have better and easier to setup and use.
Really surprised IPFS hasn't been mentioned yet. Though, it will eventually implement a cryptocurrency to help solve the hosting issue (think Storj or Sia but for websites).
But, to be brutally honest, I doubt the backbone of our internet will change at all until we have a serious, blatant cyber war wherein the vulnerable aspects of our HTTP-based internet is taken advantage of.
Peer-to-peer solutions, including IPFS for file storage/serving and various solutions proposed here for decentralized messaging, are no good for battery-powered devices, like cell phones. I don't want mine to burn battery maintaining P2P connections 24/7, or burn battery and my time re-establishing or confirming cached connections every time I wake it up. That puts us back to having a gateway or proxy, so... a server.
Content on a p2p network would have built in guranteed of immutability. This would ensure that you wouldn't need to redownload the same content over and over, nor even check to see if you need to redownload it. If the GPs suggestion is followed, the resulting network could allow you to download content from a server just as you do now, so long as you pay the server to download from it. The advantage is that this server needn't be the original host.
On the other hand, a peer-to-peer solution would mean a better chance at connectivity when you're out in the field with your battery-powered device that's already consuming power to maintain what little connectivity it can get with the nearest cell tower.
Yeah, not solving the problem really, unless you run the software locally/in a intranet.
Problem still stands, when your self-hosted instance/software goes down, I still can't communicate with people sitting literally meters away from me, connected to the same WiFi. It's incredibly stupid.
Absolutely. ad-hoc networks, using sound/bluetooth as a transport. Connected to the same network in general. We have all these things available for us to use, but we haven't so far. (well, there is experiments, but no big application like Slack has actually built something solid and usable with it)
( Technically an on-premises install solves the originally stated problem, and here the scope increased. P2P doesn't solve the real problem domain either, as soon as you aren't in the same room, a lot of the internet variables return )
It does not. My point is that me and another persons laptop could send packets to each other. But with the on-premises service, I still need to be directly connected for the client to work.
P2P networks can also work even if you're not connected to the same network. Point being that it's a direct connection to the person that you're communicating with, with no indirection.
So you really think there is a serious need for a solution that handles the 1 hour per year or less that services like slack are down? Or are you trying to solve for a location that has spotty internet? Because I have a problem with neither.
If you don't mind, can I ask what particular issues you faced? We're looking at moving from Slack to Mattermost later this year to help streamline the security clearance process so it's a little concerning to hear that it wasn't solid enough for something as simple as family communication.
I would be interested to know WHEN it was used also, because it has substantially improved to the point it is a drop-in replacement for extremely heavy Slack implementations
Happy to see patchwork and secure-scuttlebutt mentioned. These apps and the people behind them are, in my opinion, looking at decentralization in the most complete way from an application standpoint.
Now if we can combine these apps with routerless ad hoc or mesh networks, that would be a very good start towards an unstoppable communication platform.
Matrix has had a hard time during the past months as their previous funding was cut. They've been looking for new funding which is probably just around the corner. Hopefully stable funding will allow them to focus on polishing the UX
Slack seemed to be having issues with maxing out their websockets services. The problem with pure p2p for slack is the customer trust. Their customers want them to lock everything into their servers, these companies don't understand or trust p2p, if they did they could just take a look at Muaz Khan's github pages and find a way to implement their own corporate chat with audio and video purely p2p and use their own corporate coturn server to lock down access. Setting up coturn isn't easy but you can connect coturn to active directory and ldap for enterprise level authenticated access control for a p2p webrtc chat app for your own company.
Just a crazy thought, but what if Slack (just using Slack as an example) could provide a dongle or 1U server you can put in your server room which works as a proxy for Slack and fallback for clients should Slack be down. This could automatically be updated and maintained by Slack. If you have several sites and no VPN you could install this at all locations and connect them together so you can continue as normal whenever Slack is down and if Internet is down at least you can use it with everyone in your local network. Might be difficult to sync it back to Slack when everything is up and running again though ...
We really need more systems that can do P2P well (centralized is just a special case of P2P - you need a few robust, replicating nodes with high connectivity). The routing, replication and persistence of messages is completely orthogonal to the physical nodes and TCP channels. Unfortunately systems are not usually designed this way.
The problem has been solved to a large degree by some systems in the past - Croquet comes to mind, though there are probably others. IPFS is getting there is a roundabout way. Another interesting protocol/system scuttlebot.io.
I now use the same XMPP client to talk to Slack that I previously used with our own local chatrooms hosted on ejabberd.
However, others in my team are very dependent on the part of Slack that is less like XMPP or IRC: the built-in persistence that lets them scroll back or search for stuff that happened while they were away. This is clearly important to some people, who also bodged together persistence for XMPP or IRC using various bots or artfully deployed console clients in screen/tmux sessions on remote servers. I always found it strange, since I expect chat to be ephemeral.
For asynchronous communication, I wish all these folks had learned how to use email or usenet properly, since I think threaded discussion on mailing lists or newsgroups was far superior to what we have now in any of these popular replacements.
I still don't quite understand whether there is a majority opinion that "rooms", "channels", or serialized message boards are somehow a better UX for the average mind, or if it's really just the distortion of rent-seeking businesses always building this form that is easier to control and monetize for some reason.
> I now use the same XMPP client to talk to Slack that I previously used with our own local chatrooms hosted on ejabberd.
Have you had issues with your client taking a long time to negotiate a connection over Slack's XMPP gateway? I had that issue with Thunderbird's chat client and ended up having to switch over to the IRC gateway instead.
My fear is not about Slack specifically, but in general the tools we're using today. They all assume that you have a internet connection at all times, and they just break when I don't. Even though when it's just about passing a few JSON objects around, that could easily be sent locally.
Because, while you can by dint of extreme effort produce a user experience with IRC that's broadly comparable with what Slack offers out of the box, doing so requires so much relatively rare expertise, and so much more support investment, that it's not cost-effective to do so.
We have an IRC server that lives in private IP space and is only accessible by VPN. It was not rocket science to set up. If you think that running ircd-hybrid on debian requires "rare" expertise, then your definition of rare and mine do not exactly match.
Running an ircd doesn't require rare expertise. Running an IRC infrastructure that your sales team can and will use, that's as reliable as Slack is - well.
Is using an IRC client any less intuitive compared to using a locally installed Slack client (or, in the past, using something like AIM, MSN Messenger, Yahoo Messenger, Google talk, etc)?
No, as long as you pick a client that supports automatic image unfurling, file sharing via some cloud storage service or other, a broad range of emoji including customizable ones (people care more than you think!), and run a bouncer so your users don't lose their entire scrollback every time they disconnect.
I'm not saying that an IRC experience comparable to Slack can't be achieved. I'm saying that it is so much work to do so that, for many organizations, just paying Slack a few bucks a user makes a lot more sense than sinking scarce and vastly more expensive engineer time into reinventing that particular wheel.
There are a number of people that will say automatic image unfurling and file sharing are unwanted features which also pose security risks, if you want to operate a truly internal-only text chat system...
and does your IRC server / client combo work in a seamless manner with consistent user interface over practically all platforms? does it allow for chat history to be easily seen? does it allow the user experience that the non technical members of the company need to actually use it?
Yes, I do realize that. I also realize that, when you need to be able to easily include people who aren't engineers or ready to deal with what engineers deal with - that is, when you're a business - IRC requires much effort beyond the "apt-get install ircd" baseline to provide even a minimally satisfactory user experience.
Yes, at a time when Windows 98 was considered a perfectly acceptable desktop operating system, and smartphones didn't yet exist. The UX bar is much higher now.
Typing in a text box and reading text on a screen is just as easy now as it was 15 years ago. Typing and reading text on an IRC client is also no more difficult than doing so on Slack.
No we were talking about the absurd statements you made concerning IRC, that it requires "relatively rare expertise" and "extreme effort." Neither of which is true.
My last comment was pointing out the further absurd and unsubstantiated claim you made that "so many people seem to prefer Slack!" I think it's a pretty good illustration of "so many people" who don't prefer Slack.
If you're going to take part of my claim out of its context and argue against it as though it were the whole or indeed even made sense in isolation, then you're welcome, I suppose. If you decide you feel like arguing what I'm actually saying, you know where to find me.
You have made claims that are unsubstantiated such as IRC is harder to use, requires rare expertise, and "so many people prefer slack(unsubstantiated.) I have refuted all of these claims and even provided exampled of how where this isn't true. Now you want to say I took your comments out of context. I suggest maybe you should re-read the conversation. The context is textual communication tools.
P2P does not solve and cannot solve a number of problems associated with business communication.
If you are sitting in the same room with your colleagues, what do you do when there's a power outage? Sometimes there really isn't anything to do but wait. If those kinds of outages were less rare, you'd need to figure something out to smooth productivity. Whether that is buying a generator, or finding things to do regularly that don't require electricity depends on your business.
We have distributed systems that are used by normal applications to solve issues like this. When we need to, we put an ActiveDirectory replica in every satellite office. Frequently organizations forget why they spent the money on so much redundancy in the first place, re-centralize stuff, and it might be years before they experience a big enough outage that reminds them why they were "wasting" those dollars on redundant systems.
As you say, there are plenty of systems that could be made less fragile than they are. Slack is specifically interesting because they made a number of "more central" design decisions, even though chat as a protocol is easier to make "more distributed". What did they get out of this tradeoff? A ton of well packaged features and integrations with other systems.
Could the same thing be recreated in a more decentralized fashion? It is theoretically possible, but it could be years before we see a product/system anywhere near as functional. I would love to see an on-prem Slack product. Atlassian offers the closest suite I've seen that fits this bill. It is not easy to operate these systems at scale.
Are you calling my cases edge cases, or the parent commenter's cases?
There are a lot of tools that should work better offline. But a remote collaboration tool is not a good example of that. I've done work on secure private P2P messaging systems, and when you want that feature set, you really want that feature set.
But when I use Slack, it's because I don't want to have to be in the same building as all the people I'm working with. There is no way for this to gracefully degrade with a network partition. My argument is that central coordination and audit (with Slack) is not an edge case, it's the primary case.
For the box in my office most "failures" and the time to resolve them can be scheduled outside working hours. Many critical systems don't need to be up 24/7, 10/5 is more than enough. They can also be scripted with automatic failover, automated restore from backups, etc so that very little downtime happens.
With a hosted service you don't get that level of control, your software updates when the provider wants it to, not when you want.
It is a critical system where we work because we communicate worldwide often throughout the day, an outage in our messaging has a major impact. We probably wouldn't need backups (they're only useful for blame storming), but a failover or replication is important.
Unfortunately we use Skype for business. Uptime is great but other features (like sending a god damn message) are frequently broken.
Sounds idyllic, but you will struggle to make a business case unless Slack is mission-critical to your business (which it probably isn't - you can simply email, phone or walk over and talk to somebody).
Here I sit, with my colleagues, connected to the same network, with the same application open, but we cannot write to each other, because some backend server in the US have a problem.
How in the world could this not have been fixed yet? Our internet backbone addiction is seriously crippling us when things go sour. Not thinking specifically about Slack, but other more mission-critical tools that just... stops working because they cannot ask the time from their backend server.
We seriously need P2P systems (without blockchains please) that can be used by normal applications to solve issues like this. Offline and local network communication should be obligatory today.