Am I crazy to think there’s something wrong with the amount of engineering effort going into these startups? When you have engineers working into the wee hours of the morning because you don’t want a cursor to blink on a text field, I can’t help but feel that’s an enormous waste of energy and effort.
Since I first used Airbnb 7 years ago, the only real change I can think of is when they added experiences (which is awesome don’t get me wrong), surely at some point you begin to wonder if we are utilising this talent effectively.
Engineers with 5 YOE are making $500k+ a year at these companies because they are generating much more than that in revenues and profits for the companies. Airbnb is worth $110B.
It's not all blinking cursors, there are thousands of little experiments going on all the time. Increasing the number of bookings by 2% here, 3% there. Increasing the average price of bookings. Random extra features here and there to close that one whale of a client. Expanding into corporate accounts. Improving performance by 5% to take a big chunk out of the tens of millions of dollars infrastructure bill.
From a 10,000 foot view looking at the product, it always looks like it's 90% done in the first couple of years and then things just coast along lazily. But from a profit point of view, that last 10% can generate hugely outsized returns. It's the power of exponential growth, all those extra late nights and tiny improvements that move the needle just a little bit all compound over time.
From Dan Luu's excellent blog post on the value of in-house expertise [1]:
Another reason to have in-house expertise in various areas is that they easily pay for themselves, which is a special case of the generic argument that large companies should be larger than most people expect because tiny percentage gains are worth a large amount in absolute dollars. If, in the lifetime of the specialist team like the kernel team, a single person found something that persistently reduced TCO by 0.5%, that would pay for the team in perpetuity, and Twitter’s kernel team has found many such changes.
If removing a blinking cursor has a 1% chance to increase the booking rate by even 0.1%, , then it's worth it for AirBNB to pay an engineer to implement the change.
Dan Luu also elsewhere makes the argument that most of the impactful engineering work done in large companies is performed by a small percentage of the engineering group.
“most of the impactful engineering work done in large companies is performed by a small percentage of the engineering group.”
That is probably true. But we shouldn’t forget that these people can do these things often because they are supported by a large number of people who do the mundane tasks that need to be done.
I’m a cynic but I wish that were true. In my past experience it has usually been the opposite, the bulk of engineers are making unnecessary tools, re-inventing wheels, and designing systems that shouldn’t exist and serve as roadblocks to the minority of engineers who are miraculously getting things done in spite of all that.
The biggest cost of bloat comes from people being nice, it is emotionally hard to replace/rewrite code when the author is still there and not disliked by the team. Basically it sucks to make people redundant, so we try not to, and that makes it look like people aren't redundant even when they are.
For example, lets say you give a project to a competent engineer, he writes a clean and maintainable solution quickly alone. But if you schedule it to him together with another engineer of similar status but much less competent, then the other engineer will take a part of the project and basically block it since the competent engineer is unlikely to to take that fight, and instead just lets the project stall.
You don't get promoted for pointing out incompetence, there is a reason managers hires consultants to do that for them. This made me wonder, are there software consultants who act like management consultants and mostly go in and fire a lot of people? I don't think management consultants would do a good job of firing the right software people, they would need to be engineers.
Yes, but it’s done on a project basis. Kill the project, move the good people to a new project, release the others. Later, resume/restart the project.
It’s really hard to be surgical about this because who wants to be the good engineer that has to pick up the barely functioning pieces of code left behind. Who wants to reward a solid engineer with a big refactor job on an already late/failing project? The optics aren’t great. I’m not saying it never works, but as a general rule, deferring the project is often a better option.
> Dan Luu also elsewhere makes the argument that most of the impactful engineering work done in large companies is performed by a small percentage of the engineering group.
The keyword "impactful engineering" needs some clarification though.
It does not mean there's a 100x guy walking around the office while everyone is slacking off.
A specific proof of concept hacked together by a guy in a week might eventually become the company's flagship product. That's impact. However, the thing needs to be rewritten from scratch to become production ready or even deployable, and that takes far more work that does not fit the definition of "impactful".
I personally know a principal engineer of a FANG which single-handedly wrote the proof of concepts of more than a few projects that thousands of users use every single day. From his own words following one of his recent presentations, "this needs to be rewritten from scratch as this would get me rejected from our job interviews".
The 100x impact isn’t usually with proofs of concept, it’s with surgery. 1,000 lawyers would likely never identify and execute the life-saving graft, all while avoiding side effects that eventually kill the patient.
A surgical ten lines of code across 5 services can absolutely create billions of dollars out of thin air. The combination of technical, political and domain expertise required for such changes is relatively rare.
(I mean political in the purest, non-controversial sense, i.e. the communication skills to answer objections and acquire group consensus on the required change.)
I disagree for the following reason: without the proof of concept, the change in production would probably never come about. You can't really separate the impact of the proof of concept from that of the production change because they don't exist independently.
That seems consistent. If there's a 1% chance of a meaningful improvement, and 99 out of 100 changes do not meaningfully improve things, than the challenge is knowing beforehand what the most impactful engineering work is going to be.
Meta-question: does the cognizance, cohesion and awareness that goes into identifying that 1% emerge from the 99% quotient of inefficient aimless wandering, or somewhere else?
Meta-meta-question: assuming that it does, how does the line get described (let alone drawn) dividing "this work contributes to identifying the 1%" from "this is a waste of time"?
(Insert something about gradient descent and local vs global maximums here)
I'll call this a genuine question, I could definitely use some refinement of my own optimization of this problem space (and not wind up in micro-optimized dead ends etc).
And on the other side of that coin, if one were to espouse the same opinion in the context of anything even tangentially safety related and the average HNer's head would explode and they would rage click the wrongthink button.
It's all a numbers game. The median member of your kernel team, the median fire extinguisher, the median link to a sales web page, the median joist in a floor, all will do nothing of note. But if you distribute the resources properly they will hopefully be where they are needed to generate a positive ROI that pays for the overall system. If you zoom in too much or too little it all looks silly.
You need a bench. Some of the brilliant folks who are great at pushing through problems are awful and maintenance and sustainment. I worked for a bit in a SWAT engineering team tasked with addressing crisis problems or emergency response. If you don’t have people you’re developing on the bench, you won’t be able to respond to those types of things and will get bogged down with tech debt.
I was a faux “10x” person because I had license to break the rules to get shit done - because the value of what we were doing was higher than the cost of cleaning up the mess. (Not because of any brilliance on my or the teams part) We did two years worth of work in a month, but it’s still being refactored 3 years later.
What extra steps? It's literally a Pareto-type rule. He's also probably right, but not because only a small fraction of a company's engineers are any good. It's more that a lot of engineering work is done for reasons that are more speculative than people realize.
> Increasing the number of bookings by 2% here, 3% there.
I've always felt that there is a dilemma in this space. Any number of small UI features can claim a small increase that are often just noise and curated data (assuming no malicious baking), the cumulative increase isn't even close what the claims stacks up to.
The only fundamental way for a business to make more money is to invent actual, new feature and business models. Shifting a box here and there, enlarging an icon just creates a generation of glorified engineers and PMs who are placing their attention entirely on the wrong thing. Yet the success reporting often rewards this behavior and eventually drown the business in fruitless 'noise' endeavors that favors short term gains over actual innovation.
Maybe. Without outing myself, I have work on a choke point widget on the front page of a top 10 USA website. We had 30 engineers running these tests all day long. Was common to get 10 million+ increases to the bottom line on very minor changes. This was over a sample size of 100 million page views.
So a single data point? I didn't see it claimed that there is not a single small thing that can have a big effect. It's about large numbers. I think individual examples obfuscate more than they help us understand, when it is given in lieu of wider view instead of just as a support for one.
In this context, I would also like to see considerations of a larger picture. Sure, it helps any single company make more money. What about beyond that? Because money is a proximate goal, even if it's the ultimate one for a business, from a human society point of view that introduced this tool for a specific reason and not for its own sake.
For example, what does it mean to have some ultra-optimized large companies dominating their space? Would society really lose something without some micro-optimizations, even if they raise profit? The people who end up not making a purchase on a website because they are not as much caught in the optimized patterns, is it actually bad for them, or maybe they are actually better off and the optimizations tricked them into making an actually unfavorable (even if slightly) purchase? I think there is more to consider than just the view from the respective company or even the department. Are the means used to win actually good from a wider point of view that is not centered on that winner alone?
If someone does not make a purchase because something is slightly off, are they actually worse off, neutral or better in the end? It can't have been all that important to them to make that transaction, no?
Did you ever test doing nothing? I assume it's been studied, but I can imagine ways for 10% to actually be noise or seasonal/weekly/whatever variations.
> The only fundamental way for a business to make more money is to invent actual, new feature and business models.
I’m not sure this is fair for services like Airbnb where what they sell is conveniences.
Their entire product existed before Airbnb. It even existed before the internet. What Airbnb did so successful was that they branded themselves with a good product that disrupted the market by being easier to use.
The thing about disrupting markets by selling easy of use, is that your competition is already well versed in the market you disrupted. Some of them may do the dinosaur but others will take your ideas and try to make them better. If you don’t continuously improve the ease or use of your product, then people are going to download eurobooking or whatever other options there are now instead.
You know this, because you’ve likely done it yourself a million times with various applications that do the same thing, but some find a way to be easier to use.
Obviously these companies are going to want to find (or invent) new products to get included. But that is typically more of a non-software quest that eventually leads to some of the engineers being tasked with building what the marketing/sales/business process people find.
You shouldn't be downvoted for this - you're not entirely wrong. In my workplace, this kind of "bps gaming" has been a real thing the leadership has has had to take a stance on and fix with better processes and tech to measure cumulative impact.
On your second point, though, if all you focus on is building new features, you are losing out on tons of unrealised gains you could've had by making an old feature work better. At the scale a bigco works in, small percentages add up to quite a bit.
> Any number of small UI features can claim a small increase that are often just noise and curated data
The answer to this is better measurement and statistical analysis, not abandoning the effort entirely. It does take more than "haha, trend line go up" to prove a change had a positive impact, but it can be done.
One way to measure whether a team’s cumulative effect is positive is to have a long term holdout of users who never see their experiments. You can check for a statically significant difference between this holdout group and the group of users who see the features the team shipped.
You raise a point companies should take seriously (making sure the improvements are substantial) and are wrong that the only way is new features.
From a company’s perspective, the PMs and devs good at optimizing existing features already exist. Why not hire them to work alongside and behind the devs and PMs good at developing new capabilities?
> Engineers with 5 YOE are making $500k+ a year at these companies because they are generating much more than that in revenues and profits for the companies.
AirBnb lost billions of dollars last year, and it has rarely made positive quarterly earnings. Based on your example, the engineers are highly overpaid as there efforts rarely generate a profit. They are being paid by funds from speculators (stockholders and venture capital) that are betting on huge profits in the future.
In reality, these engineers are being paid due to labor market forces and extremely focused recruiting policies (eg only hiring those with degrees from top engineering schools or poaching workers from other companies that have similar policies).
it all looks nice and rosy and everybody patting their back about how impactful they are when scales come into play... and then you as a user are just wtf-ing given app because UI has been overhauled yet again, things that worked before are nowhere to be found, there is basically no migration guide.
The simple truth that most software engineers don't want to hear is - people are quite conservative in this. If it works once, most folks would be extremely happy to keep using it for next 20 years in exactly same way, as it is. Bug-less has higher priority than shiny (which is highly subjective) and constantly changing.
This is the unintuitive thing about growth stage companies and just the scale that internet companies can achieve today. They might already have a million users who are happy with how it works and would rather that the UI didn't change. But they are on track to grow to 100 million users, and so if making things a little bit more flashy improves the new user activation rate by a small percentage for the next 99 million users, that can dwarf the preferences of the existing users. Especially if the existing users grumble for a few days but then mostly continue using the product.
I believe most users see a tech product as a tool, "The thing that gets you to the thing". Whereas a majority in tech view those tools more as experiences. An experience puts more focus on the process itself, at the expense of the end result.
Unfortunately this tech oligopoly landscape doesn't leave users many ways to express this preference, but I often wonder if the culture will change at some point in the future as the industry matures.
An aging western population might have an influence here as well, in the long term. Older folks have less tolerance for having to relearn how to do the same thing they already knew how to do for the umpteenth time.
The one guy's thesis that we're wasting effort is probably correct, given the valuation is due to low interest rates. What's the Keynes quote about paying people to dig holes?
In a big company belief that something is worthwhile is better than actual value because market forces, that care about actual value, cannot act on each sub-project out of thousands of projects in the same company independently; but managers, which by definition can only care about their beliefs (idealism rears its ugly head again) do very much act on teams independently. The engineering department will pay $500k+ to engineers who exist only to pervert insufficiently representative metrics just as readily as the market would stop that from happening if they weren't hiding in an autocratic structure hundreds of times larger than the limits of their own abilities to be wasteful.
Indeed, there is no reason to think that the market has a greater ability to eliminate waste within a company worth $100B than it has within a communist country (which exists as an entity in international markets) worth the same.
There is a lot of waste in companies and there is a lot of waste in communist regimes, but they have very different root cause. Markets cannot eliminate waste fast enough, big companies take forever to react - adjust or die - and countries similarly. Also in most cases waste is not eliminated as first measure, but start with stupid cost reduction in valuable areas, while the real waste will be the last to go. Why? Because most of that waste is on purpose for political reasons in both cases - the communist party member or the nephew of the VP that occupy positions that are not needed or they don't qualify for, that is on purpose.
Also $500k engineers should have as the only purpose in life to do engineering, working on projects that are found by management to be important, not to find themselves something to do. Most engineers are bad at business decisions and it would be a waste of time and skills for them to do it anyway. An engineer with business knowledge is no longer an engineer, is a product owner or a senior architect, etc.
https://danluu.com/sounds-easy/ is a good take on this - the central example is for search, but it applies to many more products, as cactus2093's comment (and the article) both point out. (and for what it's worth, I've observed this personally too)
I think there are two levels here. You answered from a financial perspective, and in that case yes they are utilizing talent effectively, in the sense the company is more profitable. But on a larger society perspective the question is still here, is society better (large topic I know) when Airbnb increase their profitability by 2%, is it the best use society can make of these engineers?
The idea that these little experiment wins bring user satisfaction and/or revenue increase is a pure assumption.
Employees get so aligned with these experiment systems that obvious faults in statistical approaches go unnoticed. At the end of the day it becomes just a way to put something on your promotion case.
In most cases, core product stays the same over the years without significant innovation.
Talents been bought and sold to the highest bidder. its sickens me how dumb smart people can be because money. study hard, work on your talent all your life, so a large corperation can have you micro-optimise its addiction-tech to extract every last dollar for shareholders using VC captial to undercut your competitiors til they crumble and make it look like your talented engineers are doing something for society instead of just pumping and dumping.
Cancer research is hard work where smart people burn themselves out to fight for scraps. Optimizing metrics for bigtech at least easily pays for retirement. I don't know, maybe it's dumb to choose to optimize ads and get paid 500k and still be able to act like the company owes you. The alternative is to save lives, and in return live a life where you constantly have to beg for the next grant, and the people you save are only grateful to the MDs, say PhDs aren't real doctors, and vote for people who either want to cut science budgets or remove meritocracy to destroy your career.
Before you complain how people waste their talent, maybe ask yourself what you did to help them. Or broadly how to truly make society reward people who do God's work. Society cares about getting likes, don't want to pay even a little for quality work, so smart people give them a platform to earn likes and show them ads. Everyone is complicit, the worst among them put all blame on "smart people".
well most of us even the ones that get through the merit filters to work at "tech" companies are not that smart. it will take some kind of breakthrough in human cognition to make an impact on the emergent economies that result from our current neural hard-wiring.
right now these fake tech companies are literally the best we can do to keep everything moving forward on an economic basis.
Just as we tricked each other into bidding up crypto tokens, we could self-hypnotize to bid up biotech patents or something similar, but we have not yet decided to do that.
You would also see such 1% or 2% significant improvement in the AdTech being quite crucial. I worked at a local AdTech company where we were able to see real money coming in, not just metrics like CTR etc increasing.
I’m skeptical this will lead to more savings over time. I’ve been pretty much living in Airbnbs for almost three years, and the site becomes slightly more annoying over time.
Leaving a review has become a huge bloated mess. The sight is slow as heck. And driving prices up just makes me more likely to look at alternatives. It’s still good, but it feels increasingly less fast and cheap.
That said, whose life is really improved by those outsized returns from the last few percent?
Sure, the major shareholders might see their wealth increase noticeably, but anyone else (even small shareholders through option programmes) will generally not see a big effect.
Now, is that additional shareholder wealth really worth the years of engineer effort going into that?
Don't get me wrong: airbnb is an amazing service that has improved the lives of myself and other regular people.
I'm just arguing that maybe most of the benefit from society happened within the first couple of years plus maintenance and critical innovation as needed, and the rest is suboptimising for the benefit of the few.
>Now, is that additional shareholder wealth really worth the years of engineer effort going into that?
Where the engineers are partially compensated in stock, RSUs or options? Absolutely. Its not like they're focusing on optimisations at the expenses of other, more dramatic changes.
This is a great explanation and reflects my experience working at this kind of large, one product tech company. It's all about searching for growth of all kinds and cutting costs.
I get a eyelid twitch from that type of engineering. Open source teaches us one thing above all - adapt yourself a little and you gain a lot. Only develop an app to fit the specs, and the users will do the rest. It should be fine to be a little tough around the edges - consumers ought to grow slightly thicker skin.
Honestly I find the Spanish stance on this to be extremely refreshing and the only scaleable one:
- 'Good' is good enough, no need for perfect
- Doing it tomorrow is just as good as today
If you think about it, both are true. If you don't think these are true, you are far too caught up in a hamsterwheel to notice.
As systems grow, it becomes exponentially harder to do anything. What to you may sound extremely simple and easy on a small SaaS app, at Airbnb scale (after several years of development) it really requires several highly talented engineers to deliver without blowing everything up.
> When you have engineers working into the wee hours of the morning because you don’t want a cursor to blink on a text field, (...)
You're grossly misrepresenting the problems being faced and what forces engineers to work into the wee hours of the morning.
Just because stuff seems done to you or you notice no change, that does not mean nothing is being worked on.
Let's take basic A/B testing. To you, it's a blinking cursor. For the company, it's a bunch of business metrics being reported from N different components feeding into a data lake, with different versions of the same feature being deployed simultaneously to specific subsets of all customers based on their profile. But the cursor blinks differently depending on the market, and needs to comply with accessibility guidelines, on all X supported browsers regardless of their quirks. Some markets might not even have a cursor at all. Perhaps your cursor is only expected to show in a specific geographical location.
But you see a blinking cursor, and as you are oblivious to everything then you think it's just a tag somewhere. To you that's just a line of HTML, right? How hard could that be?
Imagining boatloads of complexity involved with the blinking cursors according to markets, profiles and geographic locations and whatever else really doesn't make it sound any better at all.
> Imagining boatloads of complexity involved with the blinking cursors according to markets, profiles and geographic locations and whatever else really doesn't make it sound any better at all.
These basic everyday requirements are oblivious to those who have zero first-hand contact or experience with professional user-facing problems imposed by business requirements.
However, just because you're oblivious or unfamiliar to these requirements, they don't mean they aren't requirements.
Think about it for a second. If it's necessary to target a service to specific geographical markets to comply with legal and/or business requirents, and given it's considerably more profitable to target a shop to a customer based on their personal interests, why would you ignore that and naively presume that the hypothetical "blinking cursor" is straight-forward to implement? And I'm not even touching hard technical probs which most developers aren't experienced or competent in, such as security and reliability.
There is a widespread problem in software development which is this this tendency to be very opinionated over all problems in spite of being totally ignorant and oblivious to the underlying problem domain. Everyone is an idiot except themselves, who always hold the answer in spite of not even knowingwhat the problem is, let alone understanding it.
>These basic everyday requirements are oblivious to those who have zero first-hand contact or experience with professional user-facing problems imposed by business requirements.
Could you paint me a realistic scenario for such multifaceted complexity revolving around this blinking cursor in a particular field?
I can see how this might depend on a language/writing system but kind of get lost beyond that.
I do have experience with such "professional user-facing problems imposed by business requirements" btw but they tend to be more related to desktop and factory software each used by hundreds which is probably a few zeroes less than what we're talking about here.
>And I'm not even touching hard technical probs which most developers aren't experienced or competent in, such as security and reliability.
I know more than most that software can behave in weird ways but don't you think if there's a reasonable worry that altering this blinking cursor affects security that something is off?
If you have the infrastructure to manage multiple versions and collect and analyse related information, adding/removing a blinking cursor or any other UI change and analysing the results should be trivial, because most of the process should be automated. Even in multiple territories and/or different user demos.
Unless you're reinventing the wheel for each modification.
> If you have the infrastructure to manage multiple versions and collect and analyse related information, adding/removing a blinking cursor or any other UI change and analysing the results should be trivial (...)
What leads you to believe in that? I mean, you have zero insight or understanding how things work or were designed. You have zero idea of where that blinking cursor comes from, let alone who owns that particular bit of code.
Let's think things through for a moment. Let's imagine you're talking about a blinking cursor in a random page from Google or Amazon. These are organizations where you have teams owning small widgets that show off only in specific pages, and that the page that you see in your browser come from a lengthy page engine pipeline that has all sorts of tests and failsafes, not to mention regional and localized deployments managed by whatever deployment policy.
This doesn't even take into account the whole workflow from product managers, who often demand data on the impact of touching a button.
You don't just edit a HTML file and hit save, don't you?
It wasn't my intention to dismiss the engineering challenges, I'm aware of how difficult something like this can be at a scale like Airbnb. I was more venting about the fact that all the engineering effort is directed at such a seemingly minor thing to begin with.
I do think it would be healthy for more companies / teams to ask: how would we run this if we could never hire another engineer? Or even what if we had to slowly shrink our engineering team?
I don't think engineers at AirBnb are overpaid or wasting their time but I do think most engineering leadership doesn't understand the idea of _true_ reliability where a service can run itself for long stretches of time.
Working at a startup now, it seems exactly like the thing I'd want to fix in the wee hours, because it doesn't seem important enough to put ahead of all other mission-critical tasks, yet when your brain goes into low-power mode, the healthy thing would be to step back and assess your work-life balance, and the ambitious thing is to fix another thing that requires less mental effort.
Since I first used Airbnb 7 years ago, the only real change I can think of is when they added experiences (which is awesome don’t get me wrong), surely at some point you begin to wonder if we are utilising this talent effectively.