Obviously everything I say is personal experience and opinion.
I think the layoffs were long overdue and should've happened sooner. There was a general understanding between my coworkers and I that Uber definitely over hired in 2016. Thanks to that a lot of engineers ran out of things to do which led to political infighting over roadmap, ridiculous redundant resourcing - every single team had mobile/web, severe shortage on infra teams since head count was already taken by main Uber teams. We even started building and maintaining our own chat app. I disagree that all the cool eng project that we put out and share here is a waste of time or resources. Those project was initiated by real needs on Uber and only the best make it out into the rest of the world. Engineers that shipped those projects are still here and we would've done it regardless of whether the 435 people that were hired in the first spot. I fully realize that it's an insensitive thing to say but I think it's important that it's expressed somewhere.
I think the lesson here is to grow responsibily, and it's real people's lives that are being affected. Don't hire people to alleviate your current engineer's burnout and stress. Learn to plan better and to prioritize aggressively instead of just hiring and getting everything done.
> We even started building and maintaining our own chat app
God I love the Not Invented Here disease. Somebody, somewhere, thought "hey, slack is way to expensive, we don't want to get 'locked in', and 'mumble mumble privacy cloud evil stallman' so lets write our own chat application.... how hard can it be?"
Boom, now the company is straddled by a shitty, homebrew chat application that is maintained by nobody because the original author left for greener pastures. Nobody dares replace it though because that would be sacrilege. That chat app is part of the company, after all!
Arg.... I hate, hate, hate when engineers with a bad understanding of business and too much time on their hands lock their organisations into homebrew crap that has nothing to do with the value delivered by the business.
> Somebody, somewhere, thought "hey, slack is way to expensive, we don't want to get 'locked in', and 'mumble mumble privacy cloud evil stallman' [...]
Like I completely agree with your overall point about reckless overengineering, but you also just listed three true and really good reasons to avoid integrating Slack.
I setup an enterprise-grade Mattermost server in a weekend. The hardest part wasn't even Mattermost itself, but getting SMTP working reliably in the age of DKIM/SPF/DMARC policies. This could be sidestepped by using any of the well-established email providers.
No reason to use Slack when such options exist, but also no reason to write your own chat app, either.
That kinda sucks the air out of this whole thread. "that's not entirely accurate" is a bit of an understatement. But, don't let facts get in the way of a good rant!
Mattermost has been rock solid for the last couple of years for my team BTW.
> The hardest part wasn't even Mattermost itself, but getting SMTP working reliably in the age of DKIM/SPF/DMARC policies. This could be sidestepped by using any of the well-established email providers.
This is valid not just for Mattermost but many other apps that support sending e-mails. And that's why it's important that instead of giving up and outsourcing it, we all try to set it up and have first-hand experience in challenges we have to face in the process. Because if everybody just uses Mailgun & SES, in a decade or two it won't be be possible to set up your e-mail server anymore (of course you will be able to, but your e-mails won't reach most customers).
> of course you will be able to, but your e-mails won't reach most customers
Unfortunately, I found this to already be somewhat of a reality. Even with all the best practices I went to, many MTAs simply don't trust my IP address. Most "sane" MTAs seem to greylist me rather than blacklist outright, so it just adds a bit of delay to users receiving their emails. Thankfully it seems to be gradually getting better, but gone are the days of simply pointing your apps at sendmail letting them go.
What's interesting is setting up a handler for DMARC aggregated feedback reports. I determined that it's not worth doing yourself, even Microsoft do not handle their own DMARC reports.
I ended up outsourcing DMARC report processing, if I wanted to do it in-house I'd have to setup and maintain an Elastic stack[1] which seems like huge overkill.
Let's not pretend that hosting an OSS chat server is a walk in the park for every enterprise just because you personally got one up and running in a weekend.
I'm not necessarily in the 'build one from scratch' camp either, I'm just pointing out that there's a trilemma in enterprise systems. How fast you're able to get something up and running is just one factor and not always the most important one.
Looking at this from Spain, you did do this over the weekend. I also work 6 days a week—but my hope is that this didn’t take the place of building a family, having a relationship, finding a relationship going for a hike in the mountains, citizen science, or making art. Your weekend is valuable!
You're totally right that the weekend is valuable, but some people enjoy doing this sort of thing with their free time. It's not anyone's place to decide what anyone else does for entertainment or self improvement. That is to say - what is not valuable to you might be valuable to someone else.
My comment isn't concerned with using 'personal' time for such endeavors. My concern is with the implicit conclusion that how fast you can get something up and running (a 'weekend' in this case) is an argument that you can and should deploy a new stack in any given enterprise.
You'll find a lot of times the champion of the new hotness moves on and leaves someone else to maintain the old hotness at costs that were never factored into the original deployment.
As I noted in another comment, it's a trilemma where you have to pick and choose your battles. The initial deployment time is rarely the most important factor when introducing a new stack within an organization.
When large MMO guilds have had a mostly trouble free chat client that supports a few thousand concurrent users on the free time and good will of admins I'm loathe to believe that this shit is impossible
I'm not sure how you read from my comment that I'm saying 'this shit is impossible'. I'm simply saying in many organizations 'how fast the intern got it up and running' is not always a valid indicator of how ready it is to be deployed to a few thousand concurrent users.
For example, large MMO guilds have a quite different regulatory environment from, say, a large healthcare organization in the US governed by HIPAA. It's an extreme example, sure, but a homegrown, dumbed down solution, might be the only way to ensure regulatory compliance.
It doesn't take a lot of imagination to come up with other scenarios. I can't defend Uber doing it because I'm not involved, but large MMO guilds are not archetypal of enterprise systems.
If the company relies on Mattermost, I would assume they'd need at least one maintainer per ~100 users, just to make sure that the SMTP keeps working reliably, there are backups etc. One person to maintain Mattermost would cost a company at least 150000 USD in the bay area. That's $100+ per month per user. Slack is way cheaper than that.
Really, 50 maintainers for a 5000 user system? I'd think it scales much better, probably 2-3 people for the whole company. And in that case, it can quickly be cheaper than slack
I think you are really underestimating the bureaucratic overheads in a large company. It is not the same as a 3-person startup serving 5000 customers. Of course I pulled that number out of the air, but the point is that it can make sense. I think I am right around the ballpark of needing a 5 people team to serve 500 users - you'd need a minimum of 3 people just to make sure there is enough redundancy if one person is sick or goes on vacation. You'd need 24x7 on-calls. The scaling factor probably changes after that - may be it is 10 people for 5000 users instead of 50.
Adding to other large company problems, these people will need management layers, the team will need a charter for growth. Who wants to be the maintainer of the Mattermost servers in a large company? Add all of this up and then the slack deal starts to look reasonable.
Edit: Just to add some real numbers, slack costs $12.50 per user per month [1] - that is ~$750k per year for 5000 users = 5 engineers. (More like 3 really in the Bay area)
Your general point is solid, but one quibble with the numbers - even the pricing page has an "enterprise" tier "for very large businesses".
There's no way a company with 5000 Slack users is paying the sticker price per head - they'll call Slack's enterprise sales team and negotiate some volume discount with whatever shiny enterprise-grade features, SLAs, support guarantees, etc. they need.
Lol, where did you get the idea that smtp servers need more maintenance as users increase? What if I told you I’ve witnessed a team of 3 manage a mail system of a university of over 100k active email accounts?
We are a company with around 100-150 employees, using Mattermost in a real Enterprise setting for years now (with SSO, CI and mail integration and whatnot) and that server is being maintained by our admin team basically "on the side" - nobody was hired specifically for this job, as it does take less than 20% of one person's time according to what the team says.
These Bay Area tech people must be either incredibly inefficient (especially when comparing to what they get paid), or your estimate must be waaaaay off.
From my experience that kinda makes it worse because constantly people will say "X has this now this add that" and it's normally a case of "Well that's a ton of work to do because we did Y".
It takes 1 person a few percent of their time to manage something like that for internal use only. And you still get the gains of keeping secrets internal and cost savings on licensing costs which might actually pay for a good chunk of the effort in the case of running an existing service.
Compare with a small development team for a significant percentage of their time to build a halfway decent chat tool.
Heh, this is the story of how Slack got built. Slack was built as an internal tool while building Glitch, an online game. From Wikipedia:
> Slack began as an internal tool for Stewart Butterfield's company Tiny Speck during the development of Glitch, a defunct online game.[13][14] "Slack" is an acronym for "Searchable Log of All Conversation and Knowledge.
Some day, Stewart Butterfield will launch a gaming company that is successful in the gaming space instead of becoming the inspiration for an entirely unrelated startup.
"Enterprise" slack is $15 per head per month. This would equate to an recurring annual expenditure of nearly $4.86million for Uber's 27000 employees.
Given that Slack can be seen as IRC with some pretty decoration glued up to an Elasticsearch instance, and that Uber already has a highly trained tech org in place, if it wants to make a Slack "clone" for less than, say, $25million (cost of Slack to Uber for 5 years), it could actually be a pretty sensible decision- especially when you take into account the tax and stock-price benefits of capital expenditure.
Unless NIH/buy are truly the only options. `docker run whereverthatactuallyis/mattermost` looks like a cheaper option that doesn't require building from scratch.
Not directly related to mattermost and Uber, but companies tend to list clients even if the product is not widely used or was even dumped.
You are a small team in a big company, you evaluate a product and find that it satisfies some of your basic needs but want to work with it for a while, so you purchase a couple of licenses for you team but stop using it after a year or two.
+1 for mattermost. Works out of box for most applications. But I think people will find it surprising how quickly one starts to run out of space. People can be very chatty.
As other replies have said, with a high user count, they've likely negotiated that per head figure WAY down.
But even if they didn't, it's not as simple either as "can those engineers clone this software for less than X", it's whether they can afford to take the opportunity cost of not having engineers work on something core to Uber's business.
I take it you mean Uber? I find it hard to believe Slack has 27000 employees.
Anyway, as another commenter pointed out, on an enterprise scale I'm sure you can get a good deal. I can also imagine however that at that scale you'll want multiple Slack servers / instances - which would add cost given how slack charges per person, per server.
So there's a few alternatives then; self-hosted (some parties will offer enterprise customers the option to host an instance of their application on the customers' internal hardware), or rebuild it yourself. The latter one however takes a bit of math - how much is it going to cost to build, maintain and run it? It's easily underestimated.
And yet, it's also easy for people working in an engineering culture with nearly limitless engineering capacity to just build it themselves - motivations being "how hard can it be", and "I need something more challenging than tweaking this button for my employer to earn 0.1% more money on this particular feature". I've seen it happen far too often.
Slack was evaluated a few times. The reason it was initially rejected when we first considered it is because it wasn't able to handle our scale at the time and the team over at Slack was not interested in prioritizing scaling for us over other things on their product roadmap.
Interesting. IBM adopted Slack just fine, back in 2016. I believe they've got at least 100k employees in their Slack workspace. Not sure if it's Slack or IBM doing the hosting/scaling for it, though.
There is a cultural difference between Uber and IBM which would explain why Slack didn’t shown its limits at IBM: Slack people belong to a startup which typically wouldn’t mind get everything done in a chat; whereas IBM is a 100+ years old company where everyone has to fill a paper form and managers type with two fingers (Source of this specific example: I’ve worked with IBM consultants with one of their products). Chat apps do not play the same role in each.
What about all the other chat apps? Why wasn't the whole thing dumped into a pile of hot lava the second slack could scale to an org the size of uber?
What what does scale mean anyway? Just employees using it or employees + contractors?
It's easy to armchair quarterback on this stuff, but I find it hilarious how tech companies that employee supposedly smart people justify writing this kind of stuff. I'm sure some junior engineer did a cost/benefit that completely neglected the total cost of ownership for writing a chat app and focused on the sticker shock of having to pay tens of thousands a month for a chat app.
Were these actual concerns that were raised by the actual legal council of the company, or were they issues invented by engineers who are not lawyers but suffer from engineers disease and think that knowing to code means they are experts in everything else? 'Cause I know plenty of those!
I've worked in plenty of companies with engineers who think they are legal, privacy and security experts. If we made business decisions based on these folks, we'd never do anything. Hell, every damn text change we makes invokes a few who worry about the legal ramifications ("should we file a JIRA ticket with legal?").
Sadly without somebody stepping in, people actually listen to them instead of the actual experts employed by the company... thus you wind up wasting tons of engineering bandwidth building a feature / platform / product that has no business existing all (or worse, not building some feature / platform / product) because nobody bothered to check with the legal / privacy / security team that some concern by a engineer was actually a genuine concern.
And even then, in any good organization legal / privacy / security teams exist only to point out all the risks in any business decision... sometimes it is worth the risk despite what legal / privacy / security say. What appetite the company has for the risk depends on a lot of things that are well outside of legal/privacy/security.
> Were these actual concerns that were raised by the actual legal council of the company, or were they issues invented by engineers who are not lawyers but suffer from engineers disease and think that knowing to code means they are experts in everything else? 'Cause I know plenty of those!
the general counsel and thus the entire legal org required for any company approved comms system a way to automatically by default destroy chat logs after 7 days (and to turn that off for folks who were on legal holds). slack in 2016 did not have that capability.
Not sure why the parent is down voted. I have worked in financial services and had the same experience. IT typically held a view on the interpretation of rules and regulations that was far more literal and restrictive than the actual compliance officers.
it was the privacy/legal/security types driving this strategic decision, I can't really talk about it in more depth. A lot has changed since the decision was made, and if I personally was to reassess the situation now I believe I would go with a 3rd party SAAS solution, but back then building a chat app was on balance the correct decision to make.
> it was the privacy/legal/security types driving this strategic decision
Booo on legal.... like I said earlier, it's easy to armchair quarterback. Sometimes I wonder how the fuck humanity even manages to make progress with so much waste and boneheaded decision making.
AWS is the opposite of homebrew. Lyft has no business maintaining its own data centers and operational overhead. There is a huge opportunity cost involved with running your own infrastructure. AWS and the like free your own teams from worrying about how to provision new systems, upgrading DB server versions, etc. Running your own data center is hard.
Running your own DC is the same as building your own chat app. It is almost always more expensive and almost all the people who claim they can do it cheaper inhouse aren't factoring in all the costs--both costs that are easy to measure and those that are almost impossible to measure
Examples of hard-to-quantify costs for running your own infrastructure include:
- What is the cost to the company of being three versions behind on mongo because IT doesn't have the bandwidth to upgrade? How many work-arounds do teams have to do because they couldn't use the latest features of mongo?
- What is the cost to the company when developers cannot quickly spin up new environments like they could using a service like heroku?
- How much innovation and new business (read $$$$) is not happening because the in-house IT simply doesn't have the toolset so a dev can quickly riff on an idea?
- How many engineers have good ideas and don't bother building them because spinning up an environment is too much trouble?
I'd argue by not using AWS you are holding your product teams back and stifle innovation. Unless of course you want to waste company money to build out your own half-assed provisioning tools that are lousy knock-offs of AWS's tools. But then, you might as well just use AWS...
So they're paying 26 cents per ride on all compute resources across their entire company?
That doesn't really sound that bad, considering that rides can easily cost dozens of dollars. Just about any other non-tech business would kill for such low overhead.
Other non-tech businesses, hell EVERY business that I've ever worked in that used AWS did better per transaction - healthcare, digital advertising, virtual office tooling, gaming. If you can't do better than a credit card processing fee per paid customer interaction across the company on AWS, you might as well be selling retail goods...and depending on age, your company might be soon.
Even if it is terrible, is the business such than a 26 cent overhead on rides is going to make or break it?
Companies like Lyft are doing a lot more than updating a few fields for each ride. They would be processing massive amounts of location data and they are building out models to eventually use for self driving cars.
Thank you, my point exactly. Their infrastructure costs are terrible. Especially considering how easy their data is to shard etc (buyer and seller belong are in the same geo physically).
I don't see the analogy here. This is Lyft's core business. Of course they'll go all in. On the other hand chat is not Uber's core business. They didn't have to reinvent the wheel for an internal tool.
Running your own data center is at one extreme. Most companies lease space in a datacenter, and all of the overhead of running the data center is on a different company. You'd have an infrastructure team racking and stacking, and supporting the deployments.
Uber didn't make their own chat app. They just use Mattermost. It was a comical sham internally perpetuated by the team that rolled it out that it was "built in-house".
I heard about that. That the team responsible tried to pass it off as their own. However, according to the CEO of mattermost, Uber has made extensive alterations.
Having used Mattermost a lot for the last few years, you'd really _want_ to make extensive modifications to at least the mobile clients if you want things to work properly.
Contrarian view: not your company or resources. Management was enabling it until now.
Since when did wanting to be a software developer mean there’s “one true way” to do software dev?
Why are developers supposed to be budget babysitters for a huge company with tons of people watching budgets?
I don’t owe my employer being an expert in all contexts to the benefit of their business. I am not their servant. I write code to serve contexts they want served.
Which is even more scary. Why isn't uber laying all these people off and paying cash money for a real chat application? What competitive advantage does uber have maintaining a chat application?
You can take your hands off completely, even go and do something else, read a book, write an email, book some flights, and the chat will drive itself ... somewhere.
I might actually hire an Uber for the express purpose of having someone to talk to. In Germany, taxi drivers were often former "German studies" or Philosophy students who did not find a job, so it was kind of guaranteed that you'd have a competent, honest communication partner (albeit left leaning).
I think that's brilliant. I'm curious as to why it is not regarded as proper job. I don't think it's unique to Germany though. I think it is as honorable an occupation as any. And for the philosophical types it might actually be quite a good occupation as it may not be as taxing on the mind (apart from the time they spent thinking on their favorite subjects). That frees them up to study, research and write in their spare time. Secondly I'm curious as to why they tend to be left leaning...
On what grounds would they sue? Not only is it completely legal to copy a competitor's design (in most cases), Uber isn't selling this to other companies.
I don’t understand why the above user is getting downvoted to oblivion by people who don’t understand legalities. Trade dress[1] is a form of intellectual property. This is what prevents a company from copying a competitor down to the pixel. Apple and Samsung fought an infamous and long legal battle over this. [2]
Decision to build a chat app might seem wrong now in hindsight but this could have been a strategic move (as hinted in original comment).
Look at their competitors in Asia (all-in-one services like WeChat, GoJek etc. where search, chat, ride hailing, banking etc. are all built in one app) and it would make sense for them to have an engineering team building a chat app.
Open Source chat tools are sadly almost universally disappointing. I believe I tried just about every one of them before giving up and just using Slack. I detailed the reasons here:
If you were willing to use Slack after all, then I guess you were willing to consider non-FOSS alternatives? In which case, one service you don't seem to have considered: Discord.
Yes, really, for business. Look past the theming.
Discord has a much more solid tech base (and engineering team) behind it than Slack, IMHO. A better API, too; and—unlike Slack—real support for third-party clients!
I built my own niche team-collaboration app back in 2012, and I tried out a lot of things, but ended up building my whole own stack. That was back then. If I was doing it again today? I'd just make it an alternative Discord frontend.
(There are also offerings specifically designed for the "build your own custom frontend for our chat backend" use-case, like https://pusher.com/chatkit, but IMHO Discord even has these beat.)
Right, let's just do all our business on an app that doesn't have SSO and instead anyone with the link who can sneak in has access. I'm sure that's not sketchy at all and your IT department will be thrilled.
> and instead anyone with the link who can sneak in has access
You don't have to make those links (and you can turn off, per server, the ability to create those links.) You can invite specific users. The user already has to have a Discord account, though.
And that's the thing; Discord has a different accounts model. Which is part of what I was referring to when I said they had a more solid tech base: when you're signed into a dozen Slack workspaces, the Slack client has to keep a dozen websockets open, because the open connection is for a given Slack account's view of the world, and Slack accounts are a sub-resource of Slack teams. This isn't a scalable model, and when the Slack app is using 1.2GB of memory and two full CPU cores, this is much of the reason why. When you're signed into a Discord account, on the other hand, you just have one connection to the server—and one state-sync session that includes all your open channels in all those servers—because your profile on a given Discord server is a sub-resource of your global Discord account.
It's the difference between how an email client connects to N accounts through IMAP, and how the Gmail mobile app is connected to N GSuite accounts.
And, as such, it wouldn't really make sense to have Discord SSO at the account level, because a Discord account is associated with the individual, not with a particular employee record of a particular Enterprise. It's not like an email address; it's more like a Keybase identity or a Facebook profile. (You could certainly have particular profiles under the Discord account that did SSO to a particular server, but this wouldn't be traditional SSO-as-we-know-it; it'd be more of a "connector" flow, like when an app wants to use your Github profile.)
Github is a good comparison, actually. Most corporations just use Github, not Github EE. You don't Enterprise-SSO to Github.com. Github.com is an SSO provider, in fact, that other sites (e.g. Asana) can take auth from!
I don’t think this is an advantage? Work account can be accessed by your IT department, locked out when you leave, investigated for any reason. I certainly don’t want to have any links to my personal stuff at all!
As for github.com, our official IT recommendation that people don’t use their personal accounts, but create one just for work. This way there is no worries about personal data there.
(This is at a subsidiary of a large international company. I can believe small startups use a simpler methods)
> I certainly don’t want to have any links to my personal stuff at all!
Keep in mind, putting SSO into Discord’s account system (if that ever happened) wouldn’t be like putting Enterprise MDM on your personal phone. It would be more like having Chrome Sync signed into both your personal Gmail account and your GSuite account, as separate Chrome “People” (or whatever those are called.)
With Chrome Sync, the profiles are still isolated from one another; but updates to them ride in the same sync carrier connection, because that connection is going to Google either way. In this sense, the Chrome installation itself, and its “installation-specific sync client ID”, is like a Discord account. It’s not a personal account, or an enterprise account; it’s a nothing in-and-of-itself, that has those types of objects under it.
I always thought of Electron as a way to kick out a minimum viable product quickly before writing native versions. But nope, I guess give a $16 billion company 4 years and 1600 employees and that's still not feasible.
Electron gets a lot of hate on HN. I get it. But IMHO the existence of VSC (Visual Studio Code) -- as an extraordinarily useful, powerful, compelling cross-platform application -- serves
as the only counterexample needed to puncture the "it sucks bc it's Electron" trope.
It can absolutely still be useful, and it would make sense for an early stage startup or someone's side project to use it so they're not duplicating work porting it to .NET, Cocoa and Qt or whatever. And sure, having an identical look and feel everywhere is nice. But with all the resources that Slack and Microsoft have, they need to publish a lengthy guide for the users to deal with their performance problems [1]? They take 22 seconds to load a 6 MB XML file [2]? Ehhh... it's not asking that much of a multi-billion dollar company. You deserve better!
"Visual Studio Code jumps to the end of the file quickly, but then takes many seconds for the highlighting to complete."
I'm quite fine with that, frankly. Highlighting correctly requires actually parsing the file. I use vim and the colors get wonky way too many times, because it tries to "keep it fast".
It’s not the dozen websockets in a literal sense that are the problem; they’re more an especially-visible consequence of their design choices that is easily noticeable during a design audit—a “canary” that shows the flaws of their model.
Slack has decided that each team workspace must be firewalled off from the others: such that it needs its own websocket, its own client-side sync session, its own open channels, etc. This is because Slack was first-and-foremost a web-app, and in the Slack web-app, you’d just have one browser tab per Slack team. Slack built up its infra assuming this “one tab per workspace” model, and it crept into their backend API design, and now they’re kind of stuck with it. So now, even in the native mobile app (let alone the desktop Electron app), when you have 12 Slack teams open, you have basically 12 copies of the app’s view-controllers open and idling in the background, 12 background sync controllers, etc. In the case of the Electron client, that’s also 12 renderer processes, just as if you had 12 literal browser tabs open!
You might not notice this at first, because Slack won’t initialize all those resource for a workspace’s “tab”-equivalent until you actually focus the given workspace at least once. But if you just skim through all those workspaces once in the morning, and leave Slack open, it’ll be hogging those resources all day.
(And, on mobile, that’s especially a concern, because despite what you say, web-sockets themselves have heartbeats, and 12x the heartbeats is 12x the reasons for your phone’s modem to wake up. If you’re ever wondering why the part of your phone around the antenna is getting hot even when you’re not actively using data—it’s probably that you have several active Slack workspaces in a live Slack app-container.)
>because despite what you say, web-sockets themselves have heartbeats, and 12x the heartbeats is 12x the reasons for your phone’s modem to wake up.
Don’t put words in my mouth. A heartbeat is not high resource usage.
Your wall of text doesn’t negate the fact that multiple websockets can be maintained with trivial resource consumption. It absolutely cannot be generalized to assume that multiple websockets means copies of everything. I’ve seen multiple clients supporting multiple websocket connections to distinct servers while re-using all of the same UI components.
Everyone in the Open Source community with the skills to write a good chat app is perfectly happy running IRC in a terminal. Good Open Source only happens when someone needs to scratch their own itch.
At least a year and a half ago, Android notifications weren't at all reliable, even when on wi-fi. You'd have to start the app most of the time to get them. I can't remember if it was Rocket.Chat or Mattermost that also was taking 15% of the CPU while idling on macOS, but I think it was Mattermost.
I mean, it probably wasn't in the right programming language and was missing one little feature the org thought it needed and hell.... how hard is a chat application, right?
At big companies, this is often the preferable alternative. Making and releasing quick bug fixes on internal projects is way faster than open source and public projects. Ownership and control is critical.
I recently worked with a ex-uber engineer, hired by a ex-uber engineer manager. We were piloting use of launchdarkly, and he suggested we just build it ourselves as he has already done it few times and would take him only few months to build.
I hate, hate, hate when engineers with a bad understanding of business and too much time on their hands
The equal and opposite problem is managers who don't understand the sunk cost fallacy and refuse to throw out home grown solutions. Sometimes there might have been a good idea to homebrew a thing years ago because nothing good existed. Now several better solutions exist but the company refuses to adopt them because doing so would mean "throwing away" years of work, and the managers who signed off on that work are afraid they'll look bad.
Back in 1994 I worked at British telecom on one of the first high profile web projects for our intranet.
Apparently there was a serious suggestion that this new www was shoddy and we needed to write our own version of http - this got stamped on by some senior people at Martlesham (UK version of bell labs)
> hey, slack is way to expensive, we don't want to get 'locked in', and 'mumble mumble privacy cloud evil stallman'
gets followed up immediately (almost inevitably, in my experience) with
> so lets write our own chat application.... how hard can it be?"
...rather than, say, "hey, I googled and there are a few FOSS alternatives for 'group chat with history': Zulip, for one; Mattermost, for another. Let's see if either of those fit our needs. If they don't quite, though, let's also see if either one of them has a solid, customizable codebase to fix up!"
They did. Which makes them not an example of the GP's comment. Most companies do, in fact, end up literally attempting to "write our own chat application" from the ground up. (After all, that's where Slack itself came from—NIHing an MMO-game's chat middleware!)
Is there any "chat app" that is really better than github/gitlab and a good IRC or XMPP server? I am constantly disappointed by slack. I wish google waves was still around...
I think this could be one of the reasons for Uber pulling out Twilio. Someone must have said 'hey we will build our own telephony suite .. '. I wonder how that has panned out.
> God I love the Not Invented Here disease. Somebody, somewhere, thought "hey, slack is way to expensive, we don't want to get 'locked in', and 'mumble mumble privacy cloud evil stallman' so lets write our own chat application.... how hard can it be?"
And even then they could have just used Matrix, which ticks all of those boxes.
How do you explain Amazon, then? They invented AWS to sell books on the Internet.
The real issue here is a lack of vision at the top. The current CEO needs to step down for the health of the company. He has a clear lack of vision for the company, as evidenced by his statement of "do your divisions look like what they should if we had to start over?" Who says that? Something so vague that is basically grasping at straws and shows a complete lack of strategy. Would Jeff Bezos say something like that? I highly doubt it, because he has complete control over his company and a vision that has worked at micro and macro levels.
You can do things that aren't directly related to your core business and these things should generate value both internally and externally. If you can't do this, eventually, your company will stop growing.
What does AWS have to do with an internal chat application? Is Uber going to roll out another Slack clone because they thought of something marginally better?
This where understanding business matters. AWS makes sense as a business and it benefits mostly from economies of scale which is something unique that a massive company like Amazon can provide the service which few other companies could.
NIH stuff is also rarely externalized outside of the business. Even as open source. Which as the OP pointed out needs proper investment of human capital. Not some side project of a bored engineer or two.
Most innovation happens outside of mega companies by small teams for a good reason. Even the Skunkworks type approach is only useful for certain types of work like Googles X projects. Some B2B SaaS programs roll out of parent organiations but typically they are started by teams who quit the bigger company to solve the problem they had.
You're correct that Amazon does NIH correctly and most companies do not. Amazon's pipelines and web (especially frontend) tech at scale is leagues ahead of the 'market'. However, very few people are able to appreciate this, so it's best to not even mention it on a forum like HN.
> How do you explain Amazon, then? They invented AWS to sell books on the Internet.
The first AWS service was launched over 15 years after Amazon.com started selling books on the Internet. It wasn’t even a separate brand until after Amazon S3, SQS and EC2 were launched.
> The first AWS service was launched over 15 years after Amazon.com started selling books on the Internet.
Close. It was roughly 11 years. I've been using Amazon AWS since early 2007; I believe it launched to the public in the form we know today in early 2006. They started selling books online to the public in mid 1995.
If you go back to the very first AWS services - 2002 - it's closer to being only seven years.
Amazon wasn't even using AWS at the beginning. AWS was a new product category for them. Albeit, they were able to leverage their institutional knowledge of managing data centers and probably the extra capacity of some of their servers.
In Amazon’s defense, there wasn’t anything as useful to their needs as AWS before AWS.
If a different company had dominated the on-demand computing space before Amazon’s peak load issues became too big a problem, they very may as well have gone with that and not made AWS.
Almost nobody needs to write their own chat app. Slack is fine.
That line about do they look as they should is just first line of the marketing spin they put on their layoffs. Obviously there was a clear order to cut a lot of staff to reduce payroll costs. And people here are saying there were way more staff than necessary.
Yes and no. I would love to use Slack over uChat, don't get me wrong, but uChat is a skin/integration of an enterprise chat app not a greenfield build.
That's interesting, I interviewed with Uber for an engineering management role in 2016, and was blown away when my potential boss said he had a "hiring target" for next year of close to 100 people. This was a relatively small product offshoot of Uber that is now closed. I thought that was absolutely nuts, not just the magnitude of the number but that getting bodies in seats was a metric that he/I would be measured against, but thought maybe that I just didn't understand how hypergrowth startups work and it would be a good way to gain a new perspective!
Full Disclosure: I did not get an offer, even though I walked out of that interview feeling I had never nailed an interview as much as I had that day.
Not to doubt your own experiences, but if that was honestly true then Uber should also be scaling back hiring and not be opening new major offices with large hiring sprees.
There was some earlier discussion here on why trust between employees and employers is so low now, and this is a great example. Employees being laid off while new positions open up for that exact same role so that they can hire new developers at a lower cost or avoid having to promote other employees.
This is golden. Grow responsibily. Don't overhire. Consider overhiring neglegent and treat it as such. Dissmiss managers responsible for neglegence, before or with overhires. This will also alleviate the costs of overhiring.
Over hiring is a typical mistake after raising a new round of funding. Good to hear that you agree with the layoffs. Like the old saying goes; you can't put 9 women in a room and make a baby in a month.
> Learn to plan better and prioritize aggressively instead of just hiring and getting everything done.
I often wonder how accurate the roadmap prioritization can be. I do agree it’s pretty silly to build a lot of infrastructure that will never effect your clients.
If you have for instance a billing process that needs to be automated, custom infrastructure is needed. Manual approvals of all transactions is a lot even for banks. Operations can always benefit from something that cuts their overhead time down a significant chunk. Operations approval on transactions seems like an operations problem but clients having to wait for approval means now the problem is shifted.
Hey you're still at an order of magnitude fewer layoffs than Sprint ION after they blew off a $2B loss in the early 00s. Maybe get your leadership to look at that case study.
>Obviously everything I say is personal experience and opinion. I think the layoffs were long overdue and should've happened sooner.
I assume you're not on the side that got laid off. I wonder what their opinion is...
>I think the lesson here is to grow responsibily, and it's real people's lives that are being affected. Don't hire people to alleviate your current engineer's burnout and stress.
Because those are trivial matters, and they should just suffer through them?
Seems like infrastructure/devops always gets short-staffed, presumably due to being considered a cost center. Interesting to hear someone observe this even as they're saying a company "over hired" and "engineers ran out of things to do".
Any open source project without a healthy ecosystem is always at jeopardy. Use something that doesn't require you to question NDA information from some company.
Chronosphere (https://chronosphere.io/) was recently started by ex-Uber engineers to build a commercial ecosystem around M3. M3 is Uber's metrics platform and isn't going anywhere. M3DB (the TSDB built to support M3) is becoming fairly integral to Uber in its own right. I'd say the trajectory is positive.
Actually I agree with them here, it's much more peaceful now than before. When we were trying to break into China and facing against Didi that really seemed like we were at war. Like we would start a growth campaign and merely hours after we launch, Didi would launch the exact same campaign. Even after the merger I'm not sure if this is true but we found 16 employees who were on both Ubers and Didi's payroll.
Also from the end of 2016 the company encouraged not wearing Uber swag. We had few employees, in I think the middle East, attacked by taxi drivers. They were wearing Uber swag.
Wow tell me what company you're recruiting for. I'll make sure to never apply there. This is really priveleged thinking, I've worked for Google, Amazon, and Uber, and two start ups. Let me tell you the engineering culture aren't all that different across all orgs I've worked at. It's really simplistic and naive of you to take what you hear on the media as truth. It's only "widely regarded" because Uber is under heavy scrutiny, because of the industry we operate in. Tell me what company do you work for? If all of media put a magnifying glass over you and all your coworkers, I'll gauarntee, you'll soon be widely regarded as toxic. If your company is even remotely successful, I'll gauarntee that the media will find or spin up a business malpractice of some kind.
It's really simplistic and naive of you to take what you hear on the media as truth. It's only "widely regarded" because Uber is under heavy scrutiny, because of the industry we operate in.
Right. It's not as if journalism has been a paragon of ethics and civic mindedness of late.
Hey, look, it's okay. You're most likely a good person. You're getting paid well, working on a world-changing thing, and have lots of smart people around you. Don't let one Internet comment ruffle your feathers too much. :)
Hey engineer at Uber here. I'm a male engineer started working at Uber two years ago. I'm seeing a lot of comments here that is discrediting Susan Fowler's accomplishments in taking TK down, and I would like to correct the notion that the culture/TK himself/Waymo is what brought TK down.
To preface I've work both at Google as an engineer, and frankly speaking the engineering culture at Uber didn't differ all that much. Both org had the same amount of soft misogyny, politics, and HR that protected its highest performers. In fact a lot of characters mentioned in Susan's blog post works at Google today. So what led up to the downfall of TK? Susan Fowler's blog post. What she described is what a lot of people go through in every company, but she was brave enough to post her experience using her real name. I'm willing to bet if she never posted her blog post none of this would've happened, and it would've been business as usual at Uber.
Susan Fowler exposed the weakness in our culture and it catapulted an internal French Revolution where every head had a possibility of getting axed. This put our executives in a disarray and naturally external enemies decided to strike at us when we were weakest. The Waymo lawsuit only happened when it did and only effective as it was because of the chaos Susan Fowler initiated.
So please stop discrediting Susan Fowler, what she did was really the catalyst, the trigger, for Uber to set course on a better path. It's just too bad she wasn't able to stay and see the real impact of her words. Ubers a great place to work now.
> The Waymo lawsuit only happened when it did and only effective as it was because of the chaos Susan Fowler initiated.
This seems a bit much. Why on earth would Google wait to sue? Their case was not at all dependent on internal turmoil at Uber.
Fowler deserves a lot of credit but there was significant chaos pre-blog-post (#deleteuber) and after (Greyball). Uber was having a new crisis every other week it seemed, all stemming from a cavalier, bro-y culture. That all added up, Fowler's post included, to a leadership change.
I think there's a misunderstanding of TKs character. At least internally, whenever there was a critcism externally against Uber whether it be deleteuber or greyball our engineering team only grew more determined. The more backlash from the media it made us closer together and stronger. TK was the most inspirational after external media attack and even after Waymos lawsuit he displayed vigor and confidence that we are the righteous ones and that alphabet is going to lose this battle against us. There's a misconception that these attacks weakened our culture or confidence in our leadership. It's the opposite it was a signal that we were moving fast and changing the world, and TK was at the helm defending us.
This narrative and solidarity broke down soley due to Susan Folwer. I really want to emphasize this point. Her blog post is the only piece of controversy that really made ourselves question whether we were doing the right thing. The only time we saw our great leader unable to justify our hypergrowth and the only time I personally saw him break out in tears, was the all hands discussing Susan Fowler's blog post.
So again please don't discredit Susan Folwer's blog post.
> So again please don't discredit Susan Folwer's blog post.
I didn't. I said you're over-attributing the Waymo lawsuit to it. You didn't respond to that.
As for whether Fowler was the sole factor in "TK's" ouster and none of the other crises had a material impact, well, that's a strong claim. Benchmark's letter alludes to cavalier business practices as well: "[Operating without a CFO for over two years] cannot be justified, given the scale and complexity of the business, and is symptomatic of the broader problems with past management practices."[1] There were more problems at Uber than just morale.
Wow you sound like a disciple of Kalapick's and or someone who is waiting to get rich off working at UBer via stock options or something else!
I hate Uber so much and I'm so glad to see Lyft jumping up the app store charts. Uber let 1k get stolen from me and 1000s of others and their response was basically laughing at us; customers who trusted them with their bank accounts. Their blind arrogance is disgusting and Im so glad to see it's biting them in the butt!
This crosses into personal attack, which is not allowed here. I'm sorry you had that negative experience, but please don't direct it at fellow community members.
I think that Susan Fowler is the single most influential individual in Uber's history, even more so than Travis Kalanick.
1. #deleteuber was a flash in the pan. It fizzled because it was based on a falsehood, that Uber had broken an airport strike by cancelling surge. Uber cancels surge for all notable events. Lyft somehow avoided judgement, despite also failing to honor the strike. The impetus behind the campaign was a combination of that strike action and Kalanick's presence on President Trump's ill-fated executive council.
2. Greyball is a multi-million dollar suit, it will likely be settled for less. Because Uber was only foolish enough to abuse it in a couple of cities (according to news stories and law suits, who knows what legal discovery will unearth!), the scope is limited. Furthermore, the cities that ban Uber also ban Lyft.
3. Lyft's meteoric growth in 2017 can be primarily linked to the sexual harassment stories about Uber. They even based their entire advertising campaign on it, "how you get there matters," emphasizing the moral superiority of using Lyft over Uber. The ad campaign is incredibly poignant and effective. Lyft has deftly turned the fiasco to its advantage.
Susan was able to accomplish what no person had successfully done by publishing an account on her blog of what happened. Because she had not sued the company for damages, she was not held to silence from a legal settlement. Any opponent claiming she was an opportunist had no evidence, because she didn't file suit nor was seeking damages.
Because of her, an entire chain of events was set off. The intense public pressure, that slammed customer growth, hampered recruitment, and impacted Uber investors' social lives (yes, that was a major factor!) was primarily from Fowler's revelation of the company's indifference to sexual harassment.
2. Greyball is a multi-million dollar suit, it will likely be settled for less. Because Uber was only foolish enough to abuse it in a couple of cities
For starters: It cost them their license in London. Transport for London didn't look kindly at those shenaginas and apparently didn't believe Uber that they didn't employ those dirty tricks in London.
What's the source for Uber only abusing it in a couple cities? Uber?
The problem with that is that Uber is one of the utterly least trustworthy companies in existence. I, for one, assume that the lie by default, whenever they make a statement.
> #deleteuber was a flash in the pan. It fizzled because it was based on a falsehood, that Uber had broken an airport strike by cancelling surge. Uber cancels surge for all notable events.
Not to get too caught up on this, but #deleteuber was in response to them being perceived as breaking the strike by operating at the airport at all. So the fact that they cancel surge for all events wasn't accepted as an excuse. Canceling surge just provided more fodder.
Point 3 seems to be a case where correlation does not imply causation. Basing an advertising campaign with an oblique reference to the issues at Uber* doesn't mean the campaign is effective.
* most people I talk with are only vaguely aware there are issues at Uber, including many drivers, Uber users, and non-Uber users.
There's absolutely ZERO discrediting of Susan being done in my comment that TK is responsible for his own downfall. I'm being completely misunderstood here.
My desire is simply to ascribe blame and responsibility where I believe it lies - at TK's feet.
If TK had not behaved as he had, and a. cast a blind eye to the offenses against Susan, and b. engendered a culture where others continue to deny and sweep offenses under the carpet, none of this would have happened.
Susan would not have been the victim of such awful treatment. It is so good to see her get out of there, and thrive after all that's happened!
YES it's good that she blew the whistle, but it's STILL HIS FAULT and HE is his own undoing.
"I'm seeing a lot of comments here that is discrediting Susan Fowler's accomplishments in taking TK down, and I would like to correct the notion that the culture/TK himself/Waymo is what brought TK down."
I'd like to take this opportunity to plug Joanna Russ' book, How to Suppress Women's Writing. It's hard to find now, but a very entertaining read.
Specifically, in this case, I believe this is the "she didn't really do it" step, or possibly the "but it's not really very important anyway" step.
I’m sorry, but companies’ cultures don’t change that fast and it’s not even really clear how much power Kalanick has actually ceded. Was the entire chain of command from the HR staff that tried to shut women up to the execs that passed around a rape victims medical records cleared out?
I don’t doubt that a place like Google has many of the same problems and agree that Fowler should get a great deal of credit for her courage. But actual change is a long and far more difficult process than what has happened at Uber.
> was the entire chain of command [...] cleared out?
Yes, like 80% of the C suite left. The orgs that had been the worst have had high/mid level management departures too. Our Chief People Officer Lianne has been incredible.
Male engineer (not at Uber) checking in. I find myself pretty astonished at these reports of other men's behavior. And yet, it seems to be widespread enough that some women indicate it's not as rare as we might think.
So, regarding "Uber's a great place to work now" -- is that based on testimony you've heard from women there? Because if not, I'd wonder whether you really have enough evidence to make a claim like that.
It's not really a claim, just my personal opinion. I mean there are women engineers who I talked to and they definitely most agree it's better. We all agree on one thing. Ubers a completely different company. Ask anyone that works here.
There are some who I've spoken with that told me that their previous company (think Google/Facebook/Microsoft caliber companies) which had worse culture of misogyny even compared to pre Susan Fowler Uber.
Fowler did a very brave thing. Most people who do "blow the whistle", however, end-up with nothing more than a termination and blacklisting.
People like to think that silicon valley corporations are very different than their traditional counterparts, they're NOT. HR will always, always, ALWAYS protect top-performers and the interests of the company in front of literally anything else, especially individual employees.
I think more folks should read "Corporate Confidential." Yeah, it's not backed up with "research," but the arguments are compelling and it cogently explains how Susan Fowler's situation could happen at almost any other large company.
Uber will perhaps be able to avoid another giant sexual harassment fiasco, but they're not going to fundamentally change as a result of this and neither will any other company.
The lesson you want us to take away from Fowler's victory is to accept that the status quo is impervious, to give up hope and never try?
How about we instead look at what Fowler did right so that she didn't get "termination and a blacklisting"? Then we can perhaps accelerate the progress that whistleblowers bring to industry and society.
Fowler is paying a _very_ high price for speaking out, not everyone can do that, and I don't think this should be expected of anyone. It is a sacrifice. Will her actions pave the way to make such overt sexual harassment less common in corporate environments? Perhaps.
I can tell you right now, however, that HR departments will _NEVER_ have the back of employees. They will do whatever perfunctory actions are mandated by law, but they will eject whistle-blowers and anyone else who they perceive as a "problem" at the first convenient opportunity.
What interests me the most about this entire narrative is the gentle coincidence that this all dovetails with a tit-for-tat media narrative between Google and Uber, just as the self-driving-car battle ignited between these two companies, over engineering intellectual property.
Speculation aside, observe that just as lawsuits regarding trade secrets opened up, Uber’s gasket blew with Susan’s story, and then months later James Damore’s gasket blew.
Is there a curious interpretation of this? Why are Google and Uber mentioned hand-in-hand ever since winter, 2016? Is this a question worth asking?
Obviously everything I say is personal experience and opinion. I think the layoffs were long overdue and should've happened sooner. There was a general understanding between my coworkers and I that Uber definitely over hired in 2016. Thanks to that a lot of engineers ran out of things to do which led to political infighting over roadmap, ridiculous redundant resourcing - every single team had mobile/web, severe shortage on infra teams since head count was already taken by main Uber teams. We even started building and maintaining our own chat app. I disagree that all the cool eng project that we put out and share here is a waste of time or resources. Those project was initiated by real needs on Uber and only the best make it out into the rest of the world. Engineers that shipped those projects are still here and we would've done it regardless of whether the 435 people that were hired in the first spot. I fully realize that it's an insensitive thing to say but I think it's important that it's expressed somewhere.
I think the lesson here is to grow responsibily, and it's real people's lives that are being affected. Don't hire people to alleviate your current engineer's burnout and stress. Learn to plan better and to prioritize aggressively instead of just hiring and getting everything done.