In my career I have seen almost identical systems being built by a) 6 developers in a startup in 1 year b) 100+ developers in a big company with a much much larger budget in 2 years.
Almost the same specs, but the one in the startup had better scalability and fewer serious bugs...
So, spend 10-50x less and get higher quality?
Worst is I now consider case b) quite lean and efficient compared to what I see tech consultancies doing towards public sector, banks, etc...
--
My point being: There definitely IS a room for "rockstars" (don't like that term, but interpret this as developers competent enough to carry a lot of weight on their own and work in a smaller company) to deliver a lot of value.
The problem with the model is the lack of predictability (what if the few developers you trust don't deliver), the bus factor if one of the few devs quits, etc
In a sense inefficiency is a goal, because otherwise you loose redundancy. It costs a lot to hire 100 to produce the output of 10, but much lower risk.
In my experience you can always count on a team to do in just an year what a single developer takes an entire week to finish.
What absolutely does not mean that teams are useless. But that choosing the correct size and composition for them is one of the most important decisions for a project (right there with "are we solving the correct problem?"). And "more is better" is a completely wrong way to decide on it.
Also, this doesn't make that one developer a "rockstar" or whatever else you call it.
The best teams do the most because they have the most trust. The worst teams are usually spending time paying down technical debt, or busy generating it to meet a deadline.
Same goes for "rockstars", only its easier not to "lose your groove" if there are fewer people pinging you. You still run the real risk of two "rockstars" playing out of sync, and causing a mighty mess down the line.
It's never the individuals. It's always the team culture.
I've seen 5-person teams of raw bootcamp grads run rings around conventional dev teams with a broader experience base, repeatedly. Part of it is that there are no bullshit status games going on when everyone knows everyone else is just as in the dark as they are, but (in my view) an equally large factor is that the grad teams know they're going to have to learn whatever tools the problem needs. A more tenured team is far more likely to waste time on trying to build java in python, or fighting over not being able to use their favourite libraries.
But then there's also the wider situational context: an environment that has 100+ developers in a big company will have governance tarpits and coordination overheads they almost certainly haven't measured the cost of; 10x wouldn't surprise me at all just from those two factors.
Grad teams in the same org are more likely to have standalone, lower-risk projects to work on so they're going to be more insulated from those effects.
We had a huge project with about 20 devs that went on for a year and a half. We brought in a bunch of consultants. The leads basically spent all day herding cats, and all night coding. We all agreed we probably could have done it in half the time with just the 5 leads.
This is a problem across many job functions. It's always faster for "just the pros" to bang out a project or deliverable but it's usually in the company's interest to also be training more pros.
Every more junior person on a project is going to be less efficient than a senior person but they need to level up and go through the apprenticeship phases to become a senior person.
Very small companies or teams can tactically choose to pay more and only hire top talent but most places will need to develop internal talent due to budget, recruiting, or bus factor (business resiliency) constraints.
The problem is most of these pros were high-priced consultants who left as soon as the project finished. On top of that some of the stuff they built was a huge dark gray box to us for years.
the problem is the quality of training, i can't see lots of high quality training (it exist) but isn't available like in other areas, 1 most training is buzzword centric learn this new tool, we have "methodologies" but most of them are trash, and optimize for managers convenience and feeling of control, not output, they are books out there who teach you basic of system design, and fundamentals (even this is full of bullshit like, java architects), but they are scarce and is harder for manager to justify spending in this instead of this new tool who will revolutionize the world in the next 5 year and give the manager and increase in responsibilities.
Then the non-pros never have the opportunity to learn from the pros and the pros never get a break which leads to employee morale and retention issues + tacit knowledge loss.
Pros get burnt out being constantly called on to crank out new features under pressure and never get to leverage their deep understanding in the maintenance phase.
Non-pros have no upward mobility (career progression) and get burnt out struggling to understand and maintain unfamiliar code without any guidance.
I've experienced a different situation where I was one of two pros – pillar may be more appropriate – in a team of around 15 devs – and wasn't recognized as such. I was working 70h/week on average (I never was paid for those overtime hours and my actual hourly wage was under the minimal wage), taking in charge every and any issue that may pop up, I was a reference when it came to knowing how our system worked and people would come ask me questions about the inner working of some part even when they had designed it ! I left disgruntled, and the other piller followed course because I wasn't here anymore to buffer the conflict he had with the CTO. In spite of the new hire and organizational restructuration to fill the gap, what should have taken 2 weeks to implement (a few hours to a few day in the redesign I had reimplemented at home in the weekend following my departure), took them 6 months. They had to distribute overtime hours to every dev, and in the end I received a call from the guy in charge of HR who told me the company was considering rehiring me at the condition I should behave ! Dude was fired eventually and replaced by a manager from the holding company.
As for pros needing training by other pros, I think it's more a question of culture. Once I was tasked with adapting a microservice to also run as a library, the challenge being we used JSON fields and now also had to store them in a mysql database that did not support them natively. I wrote an adapter that could have been released as a separate library because it was that generic, it took me 1 day of work, a 15 files long PR. But because it used a postwalk traversal on tiny json data triggered lazily upon accessing the specific field, cached in the ORM object for repeated access, it smelt funny to the CTO who proceeded to ditch my work away, spend the 3 following days adapting 350 files to try to get a highly specific solution then gave up, and decided in the end we'll have two separate versions of that system, in concurrent use, one as a lib and one as a microservice, hence multiplying by two the time it took to do changes to that highly repetitive codebase because "it was simpler that way".
You don't need to train pros, they have been training on their own since they are twelve.
Reasonable argument, I like it, however would add a bit of salt:
I've seen several times so far that people maintaining things are actually doing a much better job, while those who were granted honor of doing a new thing because hey they are "pros". And once they did the prototype, they are _considered_ even more "pros", because hey, they wrote it. The people who rethought and rewrote the thing later were real makers, but didn't get the appreciation of "pros".
The worst thing is that "pros" status keeps working for itself, even while delivering worse job. I wouldn't have a problem if the system could fix itself over time.
I've been on both sides of this fence. As a maintainer, you have all the time in the world to understand the system, as it is, and improve it. You have the benefits of hindsight.
As a producer of the mvp, a lot of times there is 'exploratory code' where you're not actually sure how it will work yet; or scale. So you bang out a few approaches and pick the best one for the constraints you have. And basically never touch it again.
In one case, I was working on the mvp and the team who would be maintaining it where the one's doing code review. The code turned out way better than either of us would have come up alone. It ended up being one of the coolest projects in the company and some internal secret-sauce for making money. For example, I remember one code review where they asked to add a configurable callback in. I was like 'why?' and they answered, "we're already being asked if it can do X, if you add a callback here, here and here, we can already start building that feature so you can focus on the core."
Yes, but when a "pro" takes too long to deliver something, then that thing is considered hard, because even the "pro" cannot make it quickly.
Re. scaling -- probably, agree, just I never seen myself (working in distributed systems). On my personal experience it was always more-less clear, depending on what software/database/data types we intended to use. So before implementing an MVP, I could roughly answer what price/scalability it would have. But I can see it might have, especially in fields of statistics/ML/new algorithms. E.g. "try using SIMD for this new sorting algorithm" -- only tests could show if it's any better.
But in that areas, usually managers shouldn't judge by outcome anyway, but rather by how fast they make and test hypotheses.
Wow, that blog post post was a breath of fresh air. Can you believe it was written almost six years ago? She is really seeing into the future. Unbelievable.
Maintenance benefits from "pro" skills as much as creation does; you can break something just as easily with maintenance work as with new features if you're not good at it.
To me that's a wrong way to look at software maintenance.
What do you think leads to a better result?
- I create software that I _want_ to maintain and _will_ maintain.
- I create software that I think someone else wants to maintain, I leave the project after.
At least in open source, some of the most praised and successful software was built _and_ maintained by the same people at least for a substantial amount of time.
Thinking about it, this pattern of "I, the rock star will build it, you the code monkey will maintain it." Is a red flag.
I struggle with that all the time. We have a lot of offshore people and I am pretty sure that 3 experienced people could take on a project that requires 50 offshore people and get it done in less time at higher quality.
Offshore generally means you're trying to slash costs. Say, hiring $10/day devs from India. Now, there are great devs in India, but they don't work for $10/day.
It's the same term that was used for transitioning textile and other factory work to third world sweatshops.
Hiring internationally is just that. You might get someone for a bit cheaper (somewhere in the neighborhood of 20ish percent), but that's usually in exchange for the extra hassle of managing a remote team and dealing with timezone issues. You don't usually crank through these people every quarter, they're hired for the longterm (or, as longterm as anyone in a software job is hired for).
I worked on a team with two other engineers. I was in the EU, one was in Australia, and one on the west coast of the US.
We had very, very little overlap to have meetings (unless one of us wanted to wake up early or have a late night meeting, which happened a few times). We did everything async, even meetings and decisions.
The best part would be learning how to open a WIP PR, with enough detail that someone else could understand what you were going to do. You'd wake up in the morning to find a team member had contributed to your PR, either by fixing nits (we didn't have time to go back-and-forth on shenanigans or it would take weeks just to merge a single PR), or by actually implementing some idea they had. So, if you didn't like it, you'd just remove that commit from your PR.
I had the "timezone thing" with a colleague in the same city because he had some weird sleep disorder. Basically, he was a vampire. Asleep during the day, working all night.
It was awesome. I'd turn up to the customer site to collect their issues and requests, do some investigation, formulate the precise requirements, and send it off to him.
The next morning, magically, things would be done.
When you offshore cost is the main factor and the companies optimise for it. As an example you may have an offshoring contract with Infosys where you pay around $20/hour/head and they provide mostly freshgrads (some times referred to as "commodity developers") with dubious qualifications. But you can say you have 100 engineers on the project. When hiring internationally you usually hire and interview a specific person and pay them well through Deel (or equivalent). All of the former engagements have been a disaster, the latter have mostly worked out.
Both have their benefits. In my experience offshore teams are basically an army that you need to train and provide a lot of direction. It can be very frustrating but if you have code generators, Sonarqube, linters and they are only working on CRUD applications it sometimes works out. Whereas international hires are usually very competent because you personally selected them.
To me, "offshore" means acquiring developers from international shops that bid for US contracts, whereas hiring internationally just means hiring outside the country on an individual basis.
Speaking very broadly here, my impression is that those offshore development shops bidding for US contracts often don't have the greatest working conditions. Those talented enough to know their worth and find better options will do so if the opportunity presents itself, which could mean either applying for international positions on their own or immigrating to where the jobs are. Thus there's an outflow of talent away from those offshore companies.
Offshoring is contractors. They (usually) will code exactly what you tell them, and no more. They are not "product oriented" and won't push back on things if they make no sense -- they'll just shake their heads and get it down in the fastest way possible. They are not here "for the mission" and often don't care about maintainability. It's not their fault, they know they could be let go at any time, and any personal relationships they might build in the company won't save them come budget time. So you know, they're kind of checked out. On top of that you have timezone issues and potential language barriers.
Hiring internationally, you might have TZ and language issues but none of the other stuff. If you do your hiring right that is, you'll get somebody who actually wants to be part of the team and is product oriented.
I think the poster is hinting at skill level, where "offshore" is low skill in a cheap location, and "hiring internationally" means high skill in any location (or provide visa to move to high cost location).
One thing that I have found with "offshore" hiring: The very best have _mostly_ migrated to high cost locations because they get a huge salary increase (far more than cost of living increase). There are _some_ exceptions where the candidate is very high skill, but needs to live in a developing country. Usually family or cultural reasons. However, they are very hard to find and harder to keep.
The difference is only implied in their comment. They are using "offshoring" to referring to relying on a completely outside team to deliver a product.
In comparison to hiring individuals internationally to work with an internal team.
In the first case the entire project is "offshore." In the second case only individual team members are not in the primary country.
I struggled with this a lot too. I came to the conclusion that an offshore developer who as good as a good onshore developer costs about the same, just due to supply and demand, so it's a wash. If a company can't find or retain talent, hiring a not great developer offshore is cheaper than hiring a not great developer onshore. It's an "I give up" strategy; you can build the same mediocre, buggy crap cheaper offshore. Plus, off shore sales people are very good and promise the world. Many traditional executives are gullible when it comes to technology, but they're pretty good at turd polishing, so that's also part of it too. I also think OpEx vs CapEx accounting has a lot to do with it as well.
> We had a huge project with about 20 devs that went on for a year and a half. We brought in a bunch of consultants. The leads basically spent all day herding cats, and all night coding. We all agreed we probably could have done it in half the time with just the 5 leads.
20 developers is a lot, but could in theory be worthwhile. I'd really try to get all 20 be very-effective ones and structure the responsibilities and communication so that they don't get bogged down. If you do it a naive way, and aren't lucky, it'd be easy for 20 1x..10x developers to become 0.01x developers.
Or, if you get people who just want to call themselves a "dev" and grind concrete task assignments in a straightforward way, then I think you either need to be doing something bog-standard and naturally amendable to just going through the motions (e.g., scalable Web CRUD, or yet another forum/chat app), or you need a brilliant approach to structuring the work.
> 20 developers is a lot, but could in theory be worthwhile. I'd really try to get all 20 be very-effective ones and structure the responsibilities and communication so that they don't get bogged down. If you do it a naive way, and aren't lucky, it'd be easy for 20 1x..10x developers to become 0.01x developers.
I don't think it makes sense to talk in terms of absolute numbers. It depends on the size of the company and what they're trying to build. It's more like "n good developers are faster than 10*n shit developers". If you have enough essential complexity in what you're selling to your users you could realistically have 50s or 100s of developers where every single one of them is like "startup grade" and has their own projects with their own responsibilites, their own autonomy and low communication overhead.
The problem isn't companies that have 200 devs. It's companies that have 2000 devs that could run with 200 (or companies that have 200 devs that could run with 20, or companies with 20 that get outcompeted by 2 people in a garage). I've seen all of them at some point in consulting.
The hardest thing, as noted by others, is managing the auxillary complexity -- which is often introduced by excess, sometimes underskilled, developers.
The 'good' (whatever that means) devs will waste more of their time hacking around the above... so more devs are added to compensate... repeat a few times: Fred's prophecy becomes manifest.
Consultancies could hire all the "rockstar" developers they want, but their culture is around generating billable hours, not building great software. A developer on the Scrum/Jira treadmill, working on one ticket at a time, will not have the leeway to step back and look at solutions holistically and come up with a better approach.
> In a sense inefficiency is a goal, because otherwise you loose redundancy. It costs a lot to hire 100 to produce the output of 10, but much lower risk.
The problem is the velocity hit you take by doing this is its own risk that's not tracked in the same way predictability and bus factor are. I'll bet one of the core reasons OpenAI was able to release a ChatGPT style product so much faster than Google, despite Google coming up with the fundamental Transformer breakthroughs, is this difference in velocity. And when a startup begins to erode a calcified, highly redundant larger company, then the larger company has the risk of becoming irrelevant along with all of its internal redundancy.
> The problem is the velocity hit you take by doing this is its own risk that's not tracked in the same way predictability and bus factor are.
I think there's a better way to think about risk here.
There's lower risk with the predictable team. Those are so common, especially in established business, that we can even treat this as the default state.
If you so wish, you can take on more risk and seriously slim down and get work done faster. And it will almost certainly be faster on a large enough time scale, but because it's less predictable, you're now playing schedule chicken.
The kind of risk that you bring up, risk of being less risky and going slower . . . is really just opportunity cost. And it's ok to take that on as a tradeoff for something else, too, if you do it with eyes wide open.
Risk is not a thing to be avoided. It is a thing to harness to gain an edge on competition. After all, without risk there is no profit.
I think this is more specific than just the McNamara Fallacy (thanks for the link, I knew of the fallacy but didn't know it's name.) Big tech businesses usually optimize for risk reduction in the core business. The specific untracked metric here is that of competition. Big tech seems to go around it by trying to acquire promising competitors, but with a higher anti-trust appetite in the current administration, that's becoming less effective. And even with huge acquisition offers, it just takes one breakthrough competitor to bring a calcified risk-obsessed big tech company to its knees.
>b) 100+ developers in a big company with a much much larger budget in 2 years.
Mythical man month in play. In my experience, teams working on a single project peak at around 3-4 people, depending on how well you can silo the responsibilities. You can have 200 developers on a massive project, but the planners need to be able to silo the pieces off properly and define interfaces very well early on, which requires a massive planning phase. That usually doesn't happen the way people want it to either.
Currently on my team, we have one guy responsible for the schema, back end webapis and automation. Another guy does exclusively UI, those are the two main people. We have another guy responsible for auth and misc stuff and the third guy is a full stack that came in after major development was complete who can add features pretty quickly. We have a completely separate team that handles billing operations/applications of 2 people. Our team members communicate directly with business for requirements and understanding their needs, so nothing gets convoluted passing information down through middle channels.
It's not the only way to do it, but it works pretty well for us. We write all the programming required for operations of a pretty complex medical company. This was a complete Greenfield project and we have about 10 major and minor applications so far. We also have to adapt to incoming partner companies who have their own special needs and changing regulations, so the system and team is very flexible. We're about 4 years in and we had a working system up and running from "File -> New" after about 6 months. All this during COVID.
Another problem with big companies that I've seen is that everything is done by committee and about 10 people need to sign off on any little thing. It turns into an administrative nightmare.
> In my career I have seen almost identical systems being built by a) 6 developers in a startup in 1 year b) 100+ developers in a big company with a much much larger budget in 2 years.
The Microsoft Band golfing on device UI was built by 1 developer in 2 or 3 months.
The entire notification experience was 2 or 3 developers on and off again for the product's lifecycle.
The onscreen keyboard was a couple ML engineers and a software engineer for a few months.
The runtime it ran on was ~8 engineers, including all drivers, board bring ups, custom file system, everything.
Small teams can do amazing things, but they rely on good leadership and on every member on the team being able to pull their own weight. We had 1 person design+code the entire file system, and as you mention, that put the bus factor at 1.
Me too. I've seen teams of 2 or 3 up to 10 do things in half the time of a team of 100. It's the real power of life startups. If you get the right people with the right experience in the same room and align their motivations you get amazing things like the Apple II. If you put hundreds of not the right people, but close, in the same room you get a job factory.
Perhaps some day someone will invent a way to get 20 to produce the output of 10 + redundancy...
One theory is: If you have just a little too many developers (redundancy), they become bored and find ways to introduce auxiliary complexities to keep things interesting, thus things explode to 100 developers being needed...
The result would still not be guaranteed and one of the teams would have terrible morale at the end. (The other one too probably since there's a 50 percent change they are working for nothing)
Or this isn't about "rockstars" at all but a straightforward application of the "too many cooks" principle (and its refinements via The Mythical Man Month et alia).
That is, the first project wasn't 15x as productive so much because the developers were so (intrinsically) -- or even when they were objectively all above average. But because beyond a certain optimal team size, per-developer productivity drops substantially -- even plummets precipitously (due to the factors articulated in TMM: communication, politics, etc).
Plus there's a huge survivorship / selective bias example involved here (we're not look at pairings of other projects where the smaller team failed, even with better developers, because it was just too small, etc). I also suspect that if one were on the "inside" of both projects (which of course no one was) one would see a host of other factors involved (from knowledge of the requirements to market conditions, etc) not visible to an outsider.
We can definitely see how anecdotes like these help promulgate the 10x myth at least. People hear about the fates of the respective teams and think "Of course, it's because those developers were individually that much more productive. I better start making my interview process 10x more difficult now ..."
(BTW, no need to drill down on what I just said about the "10x myth", please. It's not that awesome developers don't exist or aren't important. It's just that for a whole lot of reasons, it is both incredibly overhyped, and objectively wrong in its fundamentals -- especially when we look at the original studies whence it supposedly derived).
This is why I say "rockstar" is a synonym for mug. Why won't they pay those 6 developer 10-50x more?
I've seen it many times. Young developers working their bottoms off, creating a product that makes millions only to earn average developer salary and then being laid off when company gets new investment round.
At the same time, through progressive taxation, taxman takes (depending on the country) majority of earnings, so you won't be able to amass capital to start your own business.
This is the way how the rich captured the means of production (developers) in wage slavery.
You will make it. Screw the gate keepers and build something for you and your own people. Sounds hard because it is, but that’s life, either way it’s challenging, so keep your head up and move forward.
It's not even a matter of competency: the consultancies are there to oversell - you could have 100 great SWEs on the bank project but it's doomed all the same if the internal only SPA blog has its own nest of microservices.
As I’ve moved into bigger companies over my career, I’ve become more held back by the processes in place and I am certainly way less effective than I could be. Code reviews take longer, there are political games to play and far too many meetings, managers want you writing lots of excess documentation for documentations sake instead of coding, spending lots of time on peer review discussions of those useless documents where the reviewers nitpick just to leave comments for their performance reviews, needing to jump through multiple layers of sign-offs from stakeholders for simple changes, etc. I mean, yes, small teams without that cruft will be more effective for the most part. I will probably move back to working for smaller companies soon and it will be a relief.
This sounds like Parkinson's Law applied to software engineering organizations. I also wonder if the larger organizations actual goals have anything at all to do with efficiency in software engineering / practices. I've been inside of some very large teams (400+) and the stated goals are never the real goals, they just happen to be aligned to some degree and so it sort of works.
> In my career I have seen almost identical systems being built by a) 6 developers in a startup in 1 year b) 100+ developers in a big company with a much much larger budget in 2 years.
I spent a decade in the public sector of Denmark and I've since got into helping non-SWE startups transition into enterprise organisations as far as digitalisation goes. So I've seen quite a lot of this. I've also had the pleasure to be on basically every sides of the table, from developer, architect, manager to buyer and consultant. I think it's always down to the business processes and the management.
Sure "rockstars" are a thing in SWE, I'm frankly one of them. However I don't think the difference between a 10x and a 1x/rockstar/whatever buzzword developer is necessarily too much about our technical abilities or talent. In my opinion it's often mainly defined by your organisational prowess. If you're the type of developer who can get things done, make an impact and create business value, you're going to be given a lot of freedom and you're eventually going to settle into "rockstar" roles. This is soooo much easier in smaller organisations where the restrictions you face on how you can work are often small-non-existant. Maybe you're even going to build some of the restrictions yourself, by defining how your organisations wants to lint, which techs you want to use and so on.
As things grow, more and more people are going to enter the work processes. Some of them are going to be hired to work on the work processes themselves, and eventually you're going to end up with this quagmire of bullshit red-tape. It's there with good intentions, of course it is, nobody goes to work meaning to make the world a worse place, but all the control and the business intelligence is usually an illusion of sorts. At least in my experience. Now, if you have good managers they'll build teams and manage the processes in a way that doesn't get too much in the way, but over the years, you're going to have a lot of bad managers involved in the processes. You're likely also going to work for an organisation where upper management doesn't care about digitalisation, at least not really, despite the fact that 90% of their work force is on a computer 8 hours a day. So eventually you're going to end up with an environment where it's hard to get things done.
What is worse, often the best developers. The people who are able to deliver a lot of visible business value, are going to be plucked by management in attempts to try and use them to make other developers produce more value. Meaning that many organisations tie their best development resources up in advisory roles and waste them away in endless meetings.
So it's not that those 6 developers are necessarily better than the 100. It's that sometimes, a team of 6 developers get to do more actual work than 100 developers combined do in a year.
Obviously that's only part of it. There is a whole range of other things that impact why big organisations suck. Because the quality of the 6 man team isn't necessarily on par with the 100+ developer team if you look at the application at scale over a decade. At least not in my experience. This might not matter when you're small, but it will matter when you're IBM because a few bad years will wreck your reputation for decades.
> So, spend 10-50x less and get higher quality?
You're never going to convince businesses that it's less risky to buy from a small company. It's not like CEOs are stupid. They might not care about IT too much, and they certainly hate it when IBM fails to deliver on time and that it'll cost extra to get the thing to actually work. But they do know that it will eventually work when they go to the 100+ developer company. They don't necessarily know this when they go to the 6 man team.
In fact, one of the primary reasons you hear so much more about the 100+ developer teams is because often the small startup will simply go bankrupt when the contract fails. I'm currently working on fixing one such thing, I can't tell you much more than that, but lets just say, that the 100+ developer company would've been the wiser option.
Now, sometimes you do get the better quality, the faster deliverance and the better product form the 6 developer startup, so it's not like you're wrong either. It's just that most decision makers hates risk.
I think the "rock star" analogy did more damage to our profession than most care to admit. Rock stars get treated a certain way, they are catered to, their egos might get fluffed up, etc. None of those behaviors are compatible with working in software teams, and none of the "best" developers I know want to be treated that way, anyhow.
That said, I think this article is essentially correct. I would put it like this: building information machines out of pure information is a creative endeavor. Human performance tends to fall on a bell curve, and IMO it's true for developer output in particular. So if you want to create a hugely valuable solution to a big problem, you should probably hire the best people you can find and organize them in a way that lets them max out their creative power.
The main obstacle to this IMO is poor leadership attitudes & practices. This article's author said it best in another post:
"Boring" is a good strategy if you think the bigger existential risk for your company is that Product & Engineering will [...] fail to ship a reliable product on a reliable schedule. But if you're more afraid of the business risks of shipping average product, at an average cadence, over the course of years, you should consider deviating, at least sometimes, from the playbook whose entire definition is "optimized for safety."
> Rock stars get treated a certain way, they are catered to, their egos might get fluffed up, etc. None of those behaviors are compatible with working in software teams, and none of the "best" developers I know want to be treated that way, anyhow.
Stars are a thing, they are way beyond what the average dev can ever accomplish and they would only be held back by a team. They should be catered to and really be accountable to nobody as far as technical decisions go. Their time and energy is just too precious to be spent soothing your fragile egos. And that's not just my opinion either. That's why companies have positions like fellow, distinguished, etc.
The whole "rock-stars are toxic" is just the inability of some people to accept their limitations. Most of us are nobodies, easily replaceable code monkeys who will never accomplish anything of note. If you're of the opinion that a nice personality somehow is worth more than being excellent you are lying to yourself first and foremost. And you're not that nice anyway since you are willing to spend your time digging through stars' closets in the hope of finding any skeleton to bring them down.
Being agreeable only qualifies you to easily being worked in what are essentially sweatshops who produce crap, the subject of the article. Take note though that said sweatshops would not exist without the stars that CREATED this industry. Never in a million years, you and your team of "nice" people would have produced any of the foundations of current IT.
The rock star thing is a bit creepy.
I'd like to be good at my job but not have the job turn into my entire life. Why does the conversation around tech imply that software jobs are either grey prisons or stress balls? Methinks reality is a bit more complex, but i'm just an undergrad.
The silly part of this is that the industries he compares software to - music, television, movies - have all been skeletonized and reduced to minimal and repeatable products. Simply looking out the door should be enough to convince you that if you want to bring back the magic, asking to be treated like a television writer isn't the way.
Right now the biggest thing bringing down developer wages is the threat of being replaced with an LLM. That's not going to work out in the long run - giving developers productivity increasing tools isn't going to lessen their value. However, it's the most plausible wage suppression hustle for quite some time.
Until the limits of LLM generation get tested and felt, the downward pressure on wages is going to continue. Maybe when the first wave of LLM driven startups feel the pain of maintaining code that doesn't have an author, or maybe high profile malicious training data attacks will do it. Even afterwards, the effects of the wage curve being bent downward for a couple years are still going to be there.
> The silly part of this is
that the industries he compares software to - music, television, movies - have all been skeletonized and reduced to minimal and repeatable products. Simply looking out the door should be enough to convince you that if you want to bring back the magic, asking to be treated like a television writer isn't the way.
not really sure if this is helpful, but i see this take way too often on here. i’m convinced people who say this about the film and media industry don’t go out of their way to find good movies and listen to good music because the amount of high quality, hard to find stuff being made today is honestly amazing.
The spirit of human creativity is unstoppable and always building off itself. I’d just like to see a world where instead of being hard to find, creating the good stuff pays the rent.
Well, sure, but to the contrary: DC has had a much harder time imitating Marvel's success, even with great IP. Universal tried to create the "Dark Universe." And even Marvel is having trouble cranking out the hits with the middling success of Ant-Man Quantumania, Thor: Love and Thunder, and Wakanda Forever.
I suspect making these things more streamlined and more of a flat cultural paste through the Disney Machine is a part of the lower outcomes.
I think if we want to talk about the actual value of messy human processes, you're on the nose. The problem is that large, entrenched industries can function using a distorted, rationalized model for a lot longer than any of us can afford to just wait around.
Tbh I think it depends where you work. In 1999, if you worked at a Java shop making accounting software or whatever, it wasn't a talents/creative industry.
Same thing now days, but replace Java/accounting with React/CRUD. The rockstar code ninja 10x mentality of 2009 is maybe gone for good, but I see more crazy coders doing rockstar level stuff now than ever before. Asahi Linus and alyssa rosenzweig being great examples from the Asahi project, but these people are all over.
People talk about the complexity of the infrastructure making it impossible to do everything yourself, but in my experience is exactly the opposite. The degree of abstration, automation, and high-level tools means that you kinda can do everything without knowing too much.
>Same thing now days, but replace Java/accounting with React/CRUD. The rockstar code ninja 10x mentality of 2009 is maybe gone for good, but I see more crazy coders doing rockstar level stuff now than ever before. Asahi Linus and alyssa rosenzweig being great examples from the Asahi project, but these people are all over.
To piggyback on this, I see many coders of this level of talent in many different fields all working on their own projects that are sponsored by Patreon or similar, rather than taking traditional employment with companies in general.
I work at a FAANG company on a project from a recent acquisition. The "narrative" of the startup was much like this article describes; a few rock stars knew everything and delivered sweet features everybody loved. Well, now those rock stars are gone, and we're left to pick up the large tech debt tab left over from their free-wheeling days of fast features and relatively poor documentation, code health, and testing.
I get why this happened. A startup has to deliver features now if they want to get acquired. It's a completely different mentality from the one we have now: no worries about funding, and we want to do things right in a way that will be robust long-term. Realistically, this is how it was always going to go down.
But I can't say that this is ideal. You don't want a couple rock stars carrying your project if you also want that project to be around in 10 years and can't guarantee that all your rock stars will be dedicated for life. You need somebody else to be able to pick up the torch. You want "commodification" (although I don't think it's a very accurate term; it's actually hard to get this right, not easy, but the alternative is much worse over time).
I worked at an S&P 500 company on a project from an acquisition. It was extremely well engineered. Reliable. Attention to details that you sometimes don't see in larger companies. A lot of seasoned amazing engineers worked at that startup.
I think there are plenty of examples of amazing software coming out of small teams or even single developers and plenty of examples of large corporations producing garbage with huge teams.
> we want to do things right in a way that will be robust long-term.
If you don't have the right people that know what "right" is and have seen projects through the long term then this is often used as an excuse to over-engineer and do work nobody cares about. At least that's my experience.
Never seen a place with worst technical debt, complexity and just difficult to work in.
I am convinced that commodification is what is actually causing all the issues we are seeing.
A bunch of mediocre developers getting in the way and creating complexity without any creativity, out of the box thinking or even holistic view of the system.
But management wants more people so that everybody can leave and not holding the company hostage.
Guess what? It is only worse. Very few people can actually make progress and those people are constrained by the whole team around them.
Was the acquisition a win for the company that sold it or was it more of an aquihire deal? In the success case I think you could argue that was actually the ideal type approach for that company - focus on getting to the point where you or an aquirer can safely adopt that sort of long term mentality and clean up the codebase (or not, if it's good enough). Most projects aren't aiming to be built to last 10 years and slowing down enough to get that right might mean its a moot point if they never shipped. Having to deal with a garbage codebase for a successful product is a good problem to have because it means you have a successful product
The grass was always greener in the past / Cualquier Tiempo pasado fue mejor.
Pure creative programming work have always been a small subset of programming in general. Were COBOL programmers creative? Data Base programmers? IBM programmers? Those were the majority of software jobs in the past.
I have been mostly a pure C,Verilog, VHDL, analog digital electronics -low level engineer for a long time before becoming entrepreneur and using much higher level languages in our company. Was it creative? A lot. I interacted with machines like robots that did things like moving 20 tons or lasers or robots or whatever.
Was is painful, tedious and boring?. It was, also.
You need to make it rain using active work of whatever is necessary for your job. The fact that something is not "sexy" is a great advantage for a job as it removes most of the competition. Being hard scares most people.
The best advice I could give young people is to find hard problems and to find the techniques and tools that could make those problems easier. We use psychology and things like lisp as secret weapons.
Thinking in the 35 people that created Facebook is extremely misleading, because fb got extremely lucky, and they were hundreds of thousand of programmers working in other companies. The monetary success of fb has a lot more to do with free money created by central banks than anything the founders created as their technology was easy to replicate but its financing was not.
There are intense opportunities today like they were in the past. But they are hard, like it was hard in the past.
“Were COBOL programmers creative? Data Base programmers? IBM programmers? Those were the majority of software jobs in the past.”
Some of those jobs surely had creative aspects to them … far more creative than the recent YAWS (yet another website) using ASWF (another shiny web framework) with all the bells and whistles included.
Most new software sucks. Nobody is excited by it. Neither users, nor programmers. Rockstars don't work on boring projects.
Employers don't want rockstars, they want underpaid, overworked workers.
First web browsers were a thing. Napster was a thing. Altavista was a thing. Skype was a thing. One of the first mobile apps (anyone remembers Symbian?) was a thing.
Things that some people enjoyed using and other people enjoyed creating.
Now, what can excite you as either an user or creator? Your average LOB app? A random chat app that takes ages to start and eats all your computer memory? Uber? Netflix app?
My project manager called me a rock star one time in a kick-off meeting. My fellow programmers never let me live it down – yelling out "rock star!" when they'd see me in the halls, "come on rock star, fix your bug", "I dunno, let's ask the rock star", etc.
i hired a true rockstar once. they were the smartest developer i have ever worked with, ever. this individual could singlehandedly put together massive systems, from technical to design, with confidence and flair.
i had to let this person go because they did not know how to work with others. beyond their brilliance, they could simply not empathize with the organization's POV. everything was about them, supporting them, making them feel special, unique, and important.
this individual eventually returned the company laptop, which was bent in half. they told me that they had smoked enough DMT to understand that they had created their own god and that if my org needed further investment, to reach out to them.
i hired this person from HN "who wants to be hired." they still (to this day) add a post to the monthly thread.
The greatest developers I've known were nice people, fairly humble and reasonably good team players. I'd like to think that's more common than your example but I also think I know the type.
"his individual eventually returned the company laptop, which was bent in half. they told me that they had smoked enough DMT to understand that they had created their own god and that if my org needed further investment, to reach out to them."
I guess they were a rock star in the "Led Zepplin trashing a hotel room" sense.
I don't think you'll find metaphorical rock stars working in a team in a salaried job, any more than you'll find actual Taylor Swift doing ten hour shifts in an office building. There's nuance to that though. It's really hard to distinguish yourself working in such an environment, you don't have the time, freedom and energy to do that. And if you can't do that, you won't ever be much of a "rock star".
Not that you can't work as an independent "creative" in programming though.
I've gotten to a point where I can work on Marginalia Search full time, and live off grants and donations for at least a few years. Dunno if I'm a rock star for it, I don't even have a Wikipedia page, but I at least reckon myself a fairly successful busker.
If I don't qualify, then certainly Serenity OS Andreas who made like $300k in a week with his browser shenanigans. Him and a other people far more talented than I am are able to live this way.
But it's a weird game. I think it's more or less all about the right sort of visibility. You distinguish yourself working on some large problem in a public fashion, and people inevitably will start throwing money your way; this enables additional work and additional visibility, it gives even more opportunities.
While maybe not reaching the John Romero levels of rockstar we had in the '90s, this manner of working is certainly closer to the success of a Taylor Swift than being the most talented developer in the IT department.
my first industry job was in 1995. after the interview process the hiring manager came in with an offer. it was 4x what i was currently making. after i accepted I had a half day with the admin.
she showed me the very nice hotel that they would be putting me up at until I found a place. no hurry really. one of my new coworkers was still there after a few months.
we went back to the office for tastings. coffee. wine. beer. whiskey. it was important that they knew my tastes and would have my preferred food and drink on hand.
i know that was the point - but I certainly felt like a rock star.
I started my first industry job in 2015 well after this and I was still wined and dined the night before the interview. The cocktails were $12, I had three, my future manager assured me that there was nothing to worry about the next day. In my last job, there was still lunch at a "$$$$" restaurant (at least according to google reviews) after the on-site interview.
I once got flown out to Shanghai from NYC, all expenses paid for 3 nights to interview for the China operations of an European tech company.
I guess everything is now different with remote work but before the pandemic companies, especially your prospective manager, would go out of their way to make you feel good if they really wanted you. Those were the places you wanted to work, even if the salary was higher somewhere else, because you knew you would be privileged and respected before even stepping in the door. That meant getting more resources, having more leeway and freedom to do what you want.
I also get the feeling hackers and "creative programmers" are not as valued as much in today's market. Managers want "I" shaped as opposed to "T" shaped hires.
It was! I remember my first job in a startup in 1999, I was third employee, arriving in a 2000sqm office... There was so much money wasted, at that time the most important metric was cash burn rate, to be serious player you should be able to burn at least 1M per month, even with 3 employees.
Plus, the managers really couldn't tell who knew what they were doing and who didn't, so the money just sloshed around to everyone.
I remember a ridiculous TV commercial where a stereotypical pointy-haired boss guy was talking about hiring programmers, and he said "a candidate told me he dreams in code, and... I hired him on the spot!" And it was unironically framed as if the boss guy was doing an amazingly smart thing!
In the 1980's, the About box of most applications (Mac and DOS primarily) often listed most of the people who worked on them. Many of the people were also known to others; I usually met many at the precursor to WWDC. It was cool to see your name on something you worked on (I worked on three apps over my two startups, Trapeze, Persuasion, and Deltagraph). Some people got to be pretty rockstar-like (I remember at the first WWDC (not called that yet) in 1986 sitting on a boat with what felt like all the Mac programmers in the world, a whole bunch of us aptly listening to Silicon Beach's Charlie Jackson on how he recorded a cricket for their first game). Because apps and programmers were rare compared to today and something new, you had more of a connection to the people who built things. Today you have no clue and probably don't want to know.
Yes sir. I still remember so many DOS applications and especially games. There was a game called Cannon Fodder which had the developers as characters in the game and the About screen was so cool. Man I miss those days.
I think the problem with today's software is that there are few amateurs. Most software developers are professionals who get paid by the hour - that rather build something in 5 years then in 5 days because they get paid to do it, they do not hate complexity they embrace it. Something is taking 5 seconds you better make a framework to solve it. Software is everywhere so CEO's will keep paying. The good news is that a teenager can beat a team of 100 engineers. The large team still have an upper hand with their insane marketing budgets though. But who wants to be a developer when you can get famous on Tiktok instead.
I disagree with a lot of this for a few main reasons. And some minor ones
Application development technology has considerably improved over time - most applications simply do not need to reinvent the wheel! Yes, over-engineering by designing something that will never see more than 1qps to scale infinitely is bad - I’m sure it happens but I think it’s more a strawman. If you need a simple CRUD application with a good-enough UI you have no need to introduce additional complexity (and potentially maintenance, reliability issues) with custom tooling.
Two, the software talent market is bifurcated. There is basically commodity development of crud apps, and technically complex novel development. If you think there are no rockstars you might just be in the commodity development scene. There literally are these so-called “rockstars” being flown into SF to work on new stuff in the ML sphere or into NYC/Chicago to work on bleeding edge performance. Maybe the dissonance here is that the commodity developer market has grown a lot, and that over time some technology (like web applications - a lot harder to do at scale in 2005 vs now) shifts from rockstar to commodity as it matures.
Reverting to pets-not-cattle and statefulness can be appropriate at low scale. But honestly this is more of a “choose the right solution to the problem” thing and not a rockstar thing. Following this model as you reach large scale allows for cool production heroism and cowboy coding that makes you feel like a true hacker, but that doesn’t mean your users are getting a more reliable experience, or that your dev time is being spent efficiently.
My minor quip is that, I think as you get more experienced what you used to think of as rockstar development just looks routine because you’re better at writing software.
Another minor point: you can’t just engender a rockstar culture at a company that hires commodity developers as easily as asserted here. The big thing not mentioned: PAY. Nobody wants to get paid like a commodity developer to have to perform like a rockstar. Being a commodity developer is more chill and there is less day to day risk and stress. Once you start getting towards the bleeding edge or reinventing the wheel your work becomes riskier and requires more mental effort and attention.
> Two, the software talent market is bifurcated. There is basically commodity development of crud apps, and technically complex novel development.
I’ve been making this point for years. I think it’s telling that other nearby disciplines are bifurcated. Electrical engineers vs electricians. Architects vs builders. Etc. The virtues of a good electrician are that they’re reliable, have good customer service and they work efficiently. A good building company can bring a building up in time and in budget. But beautiful architecture work doesn’t share those values. The best architecture is bold and creative, and still has people talking years later.
I think this split already exists in programming. We just don’t admit it to ourselves. It’s different work inventing React vs using it to make a website. Inventing react, or rust, or redis requires considered boldness. It takes time to nurture the right insights to make it right. In contrast, the virtues of a good web consultancy team look more like the virtues of an electrician: Good customer service. Clear estimations. Work delivered on time & on budget.
But we’re so enamoured with the idea of ourselves as “elite hackers” that clock in / clock out programmers can’t give up the mantle. I get it. But it’s stupid. There’s no point evaluating candidates on their ability to reimplement a btree from scratch if their job is to style buttons. You’re evaluating people for a different job. Test their software estimation skills. Their social skills. Ask about workflow and productivity.
Essays like this one ask “where did all the hackers go?”. They’re still around, they’re just harder to find now. They went into weird crypto projects. Llama.cpp. Obsessive database projects like scylladb. They’re inventing rust, and finding security vulnerabilities in io_uring for that sweet Google bug bounty money. They’re in the demoscene or making programmatic art for burning man.
Do you need real hackers at your company? It depends. Are you making an apartment building (on time, on budget) or the Sydney Opera House - which is iconic now, but was almost never finished because of delays and creative disagreements? They aren’t the same kind of work.
> But we’re so enamoured with the idea of ourselves as “elite hackers” that clock in / clock out programmers can’t give up the mantle. I get it. But it’s stupid.
There aren't that many wheels that need inventing, even back in the so-called rockstar era.
> Two, the software talent market is bifurcated. There is basically commodity development of crud apps, and technically complex novel development.
This I think is the source of most of the wheel invention. Someone pays a good deal of money for 'talent' they do not actually require, and they end up working at cross purposes. Assign someone with cleverness to work on a crud app, and they're bound to try to reinvent something, if not to keep their resume fresh, at least to fight the boredom.
But it's not bifurcation, it's trifurcation. It's not build or invent, it's build, buy, or invent. We are too many of us writing applications that were never really needed in the first place. Not for any single reason, but a whole host of them, from empire building, to rent seeking from people who could have made a tool that adapted well to customers, but saw much more money in keeping them engaged with you instead of operating on their own steam. Open source is also driven by a lot of motivations, but 'your own steam' is a pretty compelling one.
This article is all over the place; I got a neck ache trying to follow it. I can't engage all the points, so I'll just address the titular point. The software industry never exhibited the most salient property of talent/creative industries: rockstar economics. And that's a good thing. Otherwise a handful of Carmacks and Deans would rake in all the money, leaving the rest to man the Genius Bar until they got their "big break".
They are, and it's reflected in the compensation. I know a developer that Meta offered ~$6 mil Total Compensation, he turned it down for $10 mil from Google.
It also exists in a much more banal state lower down the ladder. I have friends who landed in the low end of the developer market. They literally cannot do most of the work I do. They don't understand DB isolation levels, they don't have the skills to design systems around eventual consistency, they are not familiar with basic distributed system problems like clock drift, etc. I'm not even an infrastructure engineer, this is just stuff I and my coworkers need to get through our days to ship product code.
Could my friends do this stuff? If they applied themselves, sure! But they're not going to. It's also not enough to learn on the job. We occasionally hire someone who doesn't actually have the skillset and they usually flounder.
...sometimes? One of the lessons I learned from him about that level of impact is that it means the structure of your day is dictated mostly by the problem you're currently focused on. So one quarter he might spend all his time interviewing. Next quarter he might spend most of his time profiling and reading code. The quarter after he might be coding.
Next to that is a constant overhead of talks with leadership and supporting other engineering efforts throughout the company through things like doc review, chats, pairing, etc.
What he doesn't generally do is "productionize" features - code is generally exploratory in nature or fixes a serious bug (or class of bugs).
Moreover, what is he coding. Because there is zero chance in hell Google is getting a multiple of that cost back in value from him. The value for them is that he's not working for a competitor. That's it.
> Because there is zero chance in hell Google is getting a multiple of that cost back in value from him.
I have seen engineers at companies 1/1000 the size of Google generate $4 million in differentiated costs savings (definitely would not have happened without them). It seems plausible to me that someone could generate 10x that at Google.
I did generate millions in differentiated costs - multiple times. Are you looking for a badass programmer? Im based in Europe but can do remote work for whatever time zone (almost any stack). Email in bio.
If you're the sort of person who makes the team around you 2X more productive, then sure, they're getting that money back. Christ, a great engineer can make that back for a company by delivering zero code, if he can keep a team of 20 productively getting the _right_ stuff done.
While there might be cases where it truly is reflected in one's compensation, most of the time it is more like less than double what others make. And they'll be promoted to the point where they waste all of their time in meetings.
I know a dude who got a nice package from a pizza company to be CTO + lead programmer. I think he has a few people under him, but the bigger deal was that he could get a percentage of the $ saved.
He worked all day and night to get LLMs to take orders, in a month 40% of their orders are taken through the LLM. For a company with 100s of stores, this is a huge cost saving.
Guy knows how to program, he knows technology, and he knew the domain from a previous job.
I think at that insane level, they pay is no longer scaling linearly with the person's skills or productivity. $10M is on the order of 100X a normal developer salary. Is this person _really_ 100X as productive? At these levels, salary de-couples from raw skill and starts to depend more on culture checkbox things like charisma, connections, celebrity, and ability to bullshit/smoothtalk. Kind of like CEO salary. Nobody thinks the CEO is actually 8000x more productive than a normal employee. The CEO is in his position because he has the "right" pedigree and the "right" network, went to the "right" school, talks the "right" way, has the "right" ivy league mannerisms, goes to the "right" polo clubs.
You and I and probably half of HN could do the job of "CEO of a mid-size tech company." We don't, not because of lack of ability, but because we don't tick those culture/personality/background checkboxes.
You're not going to leetcode-grind or PhD your way into a $10M software engineering job.
Salary isn’t really coupled to skill at all beyond meeting some basic bar to get the job. You’re not gonna have a great career if you think the leetcode problems you can solve are how you ladder up.
> The CEO is in his position because he has the "right" pedigree and the "right" network, went to the "right" school, talks the "right" way, has the "right" ivy league mannerisms, goes to the "right" polo clubs.
None of this is true. The CEO is either significantly compensated because it’s her company (majority ownership) or because the board likes the direction the ship is being steered and wants to make sure she doesn’t leave to steer another ship.
> You and I and probably half of HN could do the job of "CEO of a mid-size tech company." We don't, not because of lack of ability, but because we don't tick those culture/personality/background checkboxes.
What is it you think the CEO does? Half of HN absolutely could not run a 2000 employee firm they are intimately familiar with, let alone an arbitrary one.
> You're not going to leetcode-grind or PhD
Leetcode no, phd yes. The fact that you put them side by side pretty strongly indicates you don’t know what compensation is tied to.Leetcode skills provide limited value to a company. A PhD with a track record of leading ML publications absolutely can put you up there.
Compensation is about what value they think they can get out of you and how much competition there is for you.
Don’t sit around telling yourself execs have trivial jobs. It just makes you look ignorant. Spend some time looking at what each brings to the table for that specific company. Barring nepotism or corruption, there is usually something significant there.
I'm no rockstar, but I've worked with people who were 1/100th as productive as me for sure. There are some real morons out there with software engineering jobs.
I'd take that deal in a heartbeat. I think I (and most other decent engineers) can probably point to tens (or hundreds) of millions of dollars in savings generated over the course of a single career.
As far as I can tell, it gets frittered away by empire-builders who take that $ and spin up new teams for nonsense. Kicking back a % to the engineers who harvested all the savings seems like a much better model.
Same! That would be a very exciting and lucrative role. Unfortunately it's basically the opposite of what the industry does today. I think we've all seen the six-figures per month AWS bills to power infrastructure for a workload that could be comfortably hosted on a Raspberry Pi.
If the industry ever goes back to caring about efficiency and cost, sign me up.
At the high end of the white collar market you're paid to deliver impact and your pay is determined by the quantity of the impact and how differentiated it is (no big bonuses for the guy who presses the "print a million dollars" button).
It's easier to prove you have a special skill. They pay you more because they can't hire a replacement. In the other case, you have to prove how much better you really are. Of course, the trouble with specialization is that the skills take time to acquire and can be commoditized or become obsolete with a paradigm shift. A more risky proposition in tech than, say, medicine.
> They pay you more because they can't hire a replacement
Sounds like the skills aren't very differentiated. IME it's quite rare for a business to miss differentiation in business impact in an order of magnitude.
Going to your earlier question, the candidate in question has many skills but is not overly specialized. He commands that price because he can point at the right place in the stack and say "There's $10 million in savings here" when everyone else misses it or "Changing the architecture in this way lets us ship 6 months faster" and he is able to do this repeatedly.
Are they? How do you know they are not 9x or 11x? It is hard to say because the industry (and I live in Silicon Valley) has no idea how to measure productivity. And this is reflected in salaries. If we could accurately measure productivity, we could compensate it, but we don't.
"Knowledge worker productivity is the biggest of the 21st century management challenges." - Peter Drucker (the founder of modern management)
Not to beat a dead horse, but is 10x about 9x or 11x? Or is it about "a magnitude more output than others" (10x)?
You don't necessarily need to quantify that exactly. If you've worked on a team with people like that, then you know it. Their managers know it too.
To your main point, the size of the markets is drastically different. There is only so much room for headline rockstars and entertainers, but there's way more room for talented technologists. Having "rockstars" in the engineering world doesn't really reduce the size of the pie for others.
If you can't measure it, it might as well be 2x or "just better".
If you can't prove your superior productivity, you are not going to get paid for it. Which is going to incentivize you to optimize for your company's promotion rubric instead, if you don't get fed up first. This leads to the lamented "promotion-driven design".
Or you can ditch the middle man and take your talents straight to the market, through consulting or incorporation.
You can definitely measure it. I've seen many times where people were struggling with problems for months with tons of operational incidents, before someone experienced came in and solved it for good in a couple of days.
So even if we believe in the hypothetical general purpose 10x programmer, a 120x one seems a little implausible, right? I think it must be a domain expert or something was really dysfunctional about the original team.
In practice one very good guy can achieve as much as a whole badly managed organization.
It's not necessarily that he's that much better technically, but he's got the right drive, focus, and business acumen, and isn't bound by methodology or legacy.
The real problem with software development is that a lot of developers are too removed from the business to deliver something practical with value in the right places.
Personally I think knowledge workers should be valued for... well.... knowledge. Turns out it was in the name all along.
The person who understands and can solve your problems effectively is more valuable than a clueless person who cannot. It doesn't matter how many hours they spend in the office. What matters is technical knowledge on the sometimes confusing topics that arise at work.
It is hard to tie every job or action to revenue. Let's say I'm a platform engineer who makes internal tools. I generate no attributable revenue of my own; I make other engineers more efficient. But how much? We can't run an experiment where engineers don't use my platform because it's mandated; directly using the PaaS is forbidden.
Or I'm a manager. I help my engineers get work done. Let's consider the happy case and assume we can measure our team's revenue. How much of that revenue is attributable to the manager's superior skill, and how much to the rockstars under him/her? What would an experiment look like; the engineers manage themselves, or we compare with the previous manager?
No company that I know goes this far. In their defense, it is a challenge. An open problem, even.
You wouldn't run an A/B test to quantify your impact, but you could pretty easily do before/after metrics (or just collect them on the way during a graduated rollout) for any feature you're rolling out. It should be straightforward to quantify productivity benefit for a large initiative.
> How much of that revenue is attributable to the manager's superior skill
The output of a manager is the output of their team. We can't A/B test managers (or at least, we probably shouldn't), but we can look at a cohort of managers and see how well their teams are generating business impact. Combined with other data (e.g., upward feedback surveys, level-skip 1on1s, retention metrics, etc.), we can measure the impact of a manager.
> The output of a manager is the output of their team.
People say that but it makes no sense. That means the same manager would have a different "output" going from one team to another. Are you going to cut his/her salary if he leaves a big, successful team to help fix a flailing team because it has less output?
I get it: doing it this way is easy. My ideal is marginal attributable revenue or profit. How much better are you than the next person? We can haggle over the marginal part but I'm not conceding on attribution. Otherwise you're mooching off other people's work. Why are we paying you the big bucks? Show me what you did.
> People say that but it makes no sense. [...] Are you going to cut his/her salary if he leaves a big, successful team to help fix a flailing team because it has less output?
I think it works fine if we add on the concept that a good manager operates on a different timescale. The on-boarding timeline for a first-line EM can be months at some companies.
In your example, the manager moving over should be given a compensation package based on the expected value he's going to get out of the flailing team once he works his magic. (And, indeed, this is exactly the kind of package often given to externally-hired CEOs brought in for turnarounds.)
> Otherwise you're mooching off other people's work. [...] Show me what you did.
But that's just it. The manager's skill is improved utilization of resources. The leader's skill is improved direction of resource utilization. By definition, those skills are expressed on the canvas of the team they lead.
Or, more simply: A leader without followers isn't actually a leader at all!
And, indeed, we see that in the real world too: A successful executive in one company goes elsewhere, doesn't fit in with the new company's culture, and performs terribly. It's a tale as old as time.
Managers need to sign off on how many hours it saves. Its their neck at the end of the day.
It helps that we are often saving ~400 hours/yr spending ~80 hours or less.
We can also see it in overtime hours being eliminated, and the (real) engineers are making more complex designs but the size of the team hasn't changed.(30% more complex in 2.5 years, same team size) They also havent replaced engineers that left the team.
Basically: We save enough hours that it is obvious. There are added benefits like not having humans do manual checks, so less prone to errors. (Although I argue that our bugs are equally as dangerous)
Making sure those $ saved didn’t screw things up somehow is the hard part to measure.
And - this is going to blow your mind - but sometimes you can actually make things more efficient by spending more money. Cutting costs can be the exact opposite of the right thing to do.
So while I am jealous that your productivity can be measured so simply, I’m dubious about how many programmers’ contributions can be so easily reduced to a simple metric.
The playbook works and the more boring the better. If you are doing something on the edges of computing or industry than hooray - you are off the beaten path and it's your time to shine! You can be the one writing the playbook in a retrospective, but please don't try to make a boring application in a boring industry exciting by being unorthodox. I don't want to be excited and interested in how you solved a basic CRUD implementation - I want to intuitively know what it's doing before I even git clone your repository.
I do think that as more things become programmable and hence part of software, the odds that a given project is novel increases. If you maximize the Lisp/hacker/Dry approach, any new code should be novel; if something is just a repeat, you should make an abstraction to make it the same; instead of hand made websites just configure your Wordpress or Django or whatever.
The thing I find most sad about articles like this is that it doesn't seem to actually address any of the reasons that it got this way, it blames individuals within the field not a series of MBA graduates telling you what the spec is and hiring 50 people to hit an arbitrary deadline for a software project moving in the wrong direction FAST.
It's a false dichotomy to say you only have rock stars and as this person smugly tip toes around "normal people" when in reality you don't need rock stars anymore to make good software and let's be honest... Most rock stars didn't make good software they just make it in a time when software was generally even more crap than it is now.
You want to stop suffering among us plebs? Don't advocate for goofy rockstar developer propaganda, advocate for healthy work life balance and reasonable deadlines for things that truly don't matter. Stop letting sales and marketing write your software and stop taking opinions about systems design from your project managers and "technical leads" when they do not work in these systems day to day.
If you treat engineers well and respect them before a client who will drop you the moment a new product fits their need then yes you will lose clients from time to time but if you focus on making good software and happy people then you will attract stable clients who do the same and maybe the stock holders at the top don't get the ridiculous return per year that they expect out of more shameless companies but at least you have a half decent chance of sleeping at night...
I am well aware that we live in a world where this will be borderline impossible but the first step to solving a problem is admitting it
Not sure If I agree with this article. My experience in the big co is that trying to navigate through large systems with many moving parts and stakeholders and come up with a solution and execute as quickly as you can is very exciting. And you definitely need to be creative to trying to align and solve some many problems in 1 design. It's a very challenging and fun experience
Not to bring in this separate issue, but it can be impossible to navigate this as a technically skilled minority . Some people just don't want to work with you and are not going to work with you unless they can tell you exactly how things should be. In large orgs, you may end up with more than one of these people and they just don't align. If coming up with a design that pleases both of them is possible at all, it ends up being a turd architecturally . Later, because its a turd, people pin blame on you.
When you are blocked by non technical issues, it should be the manager's job to help you clear the roadblocks. If the manager is not doing his job, it's probably time to switch teams or jobs. Tbf haven't seen anything like this yet. There are tons minorities working in tech(myself included)
I did change jobs a few times but only for the compensation reasons. My current manager and the past managers are great.
Expectations goes both ways. The companies expect the ICs to ship code, and the ICs can expect to work with competent managers. Pay attention when selecting your team.
Eh...not so much. There are definitely good managers out there. If you're a strong IC and you have reasonable expectations on what you need for support, it's usually not _that_ hard to identify a good manager when interviewing.
These industries just have a different mechanism, here is an example: the worlds 10,000th best tennis player probably makes $0 dollar from tennis, and the world's top 0.1% of pharmacists make maybe twice as much as the 50th percentile.
Some industries you either are the top 0.01% and make millions; in some industries being average means a decent living.
Software has long transitioned from one end of that spectrum to something more towards the middle. Super star developers simply aren't productive enough for the demand of software
To be correct, even the worlds top 50th best tennis player makes $0 (read from couple interviews back in the '00s), it's mostly their family that pays. Below top 20, sponsors may cover part of costs.
Like other sports, it's fun enough that many play it for free at local tournaments, and some families can afford pay expensive flights and stay abroad.
I recall meeting with sportsmen and travellers who do months-long trips across continents. Half of them were pensioneers and just one single guy was a 34-yo die-hard traveller riding a bike across Eurasia for its own sake. Others were professionals, and they'd say they'd been preparing the travel for 2 years, seeking sponsors -- some in equipment, some in money, -- and signing contracts and clarifying sponsorship activity details -- photos here, report and booklet there, etc.
This contrasted a lot with our IT industry "let's do it for fun or learning" spirit.
It's a bit scary how much weight is put upon k8s and terraform and how little actual skills these jobs require. I miss being a sysadmin in this commodity world.
I struggle to understand why declarative tools are apparently so much more difficult to use than imperative. Why is it my team can, with great mediocrity, write a shell script, but cannot write a terraform or ansible to do the same?
Why is it they can follow a recipe to kind of make a mint birthday cake but they can't point at a picture of a cake on a picture menu and say "that one!"
Sometimes I think they don't actually know what they want before they start. And as the script evolves, like finger painting, they start to kind of like the blobs ...
But, I dont work at a software company. I just work at an enterprise with a large IT department.
>Why is it they can follow a recipe to kind of make a mint birthday cake but they can't point at a picture of a cake on a picture menu and say "that one!"
They point at the picture of the birthday cake, and they receive an apple pie and cannot figure out why. You have to read the documentation to understand that standing on one foot while pointing makes a difference.
Declarative tools are easier when you understand the domain. Otherwise, problems arise because small changes in the requirements can result in huge changes in the outcome.
> Why is it my team can, with great mediocrity, write a shell script, but cannot write a terraform or ansible to do the same?
They could. They just don't want to, because it's not as sexy as writing code. If you can't do it in Python, from scratch (lol with a lot of modules and very little exception handling), it's clearly inferior and a waste of time to learn. Their intent is to write code, not get things done!
They don't learn how to write shell scripts either. I'm sure they will go their entire lives without ever reading the man page. But will read a million bad blog posts about Python and take from them the worst conventions from mediocre programmers trying to get eyeballs on ads.
One interesting thing to note: whether you think software should be more like a “talent/creatives” industry, or more like a trade, one thing both of those models have in common is unions.
software as auteur work succeeds when it's entering a vacuum. if you're legit breaking open a field that's never been exposed to it before, sure, go wild. those fields are fewer and far between nowadays
unlike creative industries, there's a fuckton of toil work that just needs to be done automating processes. the playbook style excels there, and unlike artistic work there's not really room for memorable standouts that break the rules.
music will always have the quality where good novelty is impressive and breaks boundaries. business processes less so. prince can (could) deliver a stellar rendition of purple rain that breaks the bonds between heaven and earth even though you've heard the song before, but ain't nobody delivering a stellar automated check deposit workflow execution that stands out from the rest.
commodity services bring commodity work. this is fine. i do not want my mobile check deposit to be a tour de force, i want it to be something i don't give a second thought. there is beauty in the blase' being blase'
The transition from a new growth industry to a mature industry under conditions of nearly interest free loans and low aggregate consumer demand. In this essay, it seems clear that the kinds of designs matter a bit, but lots of companies choose different technologies and practices and survive.
The obvious conclusion is that market conditions matter more than the technology stack for a space of "reasonable" choices about it. I expect, until the next big thing hits, the market conditions to be dominated by consolidation. Bitcoin was supposed to bit it, but that was a scam. VR was also a scam (Zuck tried to sell us non-existent real estate), but maybe it'll turn into something more useful soon.
It still is, and like actual "rockstars", the workers are being exploited by ownership that, for some reason, are the only ones who are supposed to profit off of "rock".
Don't know how many musicians you know, just ask one...
Software was never really a "creatives" industry, in the way people and articles like this mean. It was, at the dawn of computing, the realm of typists and mathematically inclined women, then slowly through the 70s and 80s, increasingly nerdy men, including the "rockstar" neckbearded ingenious tricksters finding incredibly convoluted ways to save 500 bytes. Besides being "wizards" of a world most people could never understand, you had to actually bend your brain to find solutions to hard problems, because technology was so limited.
Now there are no hard problems. Just throw more hardware at it. Just write another CRUD app. Just wrap it in JavaScript. It is commodified because it's now a commodity. Software is ubiquitous and easy. There is no need to be creative, any more than for filing your taxes.
The people who like to program are, almost universally, nerds who get off on logic, solving problems, building things. But there aren't really new problems to solve. So they try to re-solve the same problems, over and over, without improving on what came before. Never satisfied. Software will always be the realm of these people that are obsessed with reinventing their toys in a sandbox. So the products will always be kind of toy-like.
That's why hardware gets better while software stays about the same. Hardware people can't just play with toys. If the hardware doesn't get better, nobody will buy new hardware, and they'll be out of a job. But people will buy different software and call that "better". Even though it's doing about the same thing as before, less efficiently.
To me writing software is similar to writing fantasy. You need to find good abstract concepts and how to compose them. Then you need to take care of how you put them in whatever language you use. I find it to be a very creative process and I pity the one who does not find their creativity at least tickled when writing code.
Writing fantasy should require what you write to be good, or at least entertaining, to sell; right? But bad fiction sells almost as well as good. You just need to follow a formula to put together the rudiments of something functional, and then find somebody who's good at selling. Slap a good cover on it, buy a good blurb, and get it in the grocery store.
Nothing personal, but you are who I'm talking about when I say nerds who just like to build toys in the sandbox. I don't think you intend on improving technology, or making a better product, as much as chasing that tickled feeling of sitting and writing. It's just amazing to me that people still get paid very well for effectively writing a copy of a copy of a book, while most fiction writers barely scrape by. You'd think the people paying us would just use the last book and not pay you to write it again, or demand it be better. But we're all fine with photocopied monotony.
Creativity is tying one hand behind your back and assembling IKEA furniture. Or making a brand new film that's nothing like the old films, that says something new, tells a new story. Rewriting an old story with new words, using the same old tropes, will sell. But it isn't creative.
Apologies in advance for being a bit negative. I feel like this sentence is just so revealing:
> That said, he and I will likely never see eye-to-eye on this because he more-or-less invented Choose Boring Technology, and most of what I advocate is to break playbook, embrace narrative, and be interesting.
It really comes down to what you prioritize. Do you prioritize the output and the value added? Or do you prioritize making your engineers feel like "rockstars"? If you don't care about your end product, then yeah sure let your engineers break prod using untested technologies.
The reason playbooks have sprung up, is because 99% of use-cases for businesses are for the most part "solved." We're not in the era of renting colos and manually setting up MySQL anymore (which the author advocates for in order to return to "creatives," btw). We have managed solutions that you can throw money at. So, do you leverage those solutions to help you solve your problem? Or do you let your engineers re-invent the wheel so they can feel smart?
The "rockstar" treatment is a result of supply and demand. Talent in Hollywood/music can get the rockstar treatment, because the talent alone is the difference between millions of dollars. If you have a really talented coder who can be interchanged with another really talented coder, you do not have a "rockstar."
The "rockstar" era of coding was the result of a massive imbalance in supply/demand for engineers, so the rockstar treatment would help attract talent. Now that the supply has began shifting up to meet the demand, it makes sense that the "rockstar" treatment is starting to go away.
And good riddance to all that too. I don't need to work with any more software engineers who think it's okay to be a massive dick to everyone just because they can implement a CRUD app. If you want to be a rockstar, go actually be a rockstar, don't make software engineering worse.
> It really comes down to what you prioritize. Do you prioritize the output and the value added? Or do you prioritize making your engineers feel like "rockstars"? If you don't care about your end product, then yeah sure let your engineers break prod using untested technologies.
First, the post is about sentiment. It's a response to a piece called "Software and its Discontents," about how people feel. So it's going to focus on that.
But I also think the narrative of people working on a project does impact its success. I don't think you pick EITHER "make developers feel like they're doing great, innovative work" OR "do you care about the end product for customers"; the key is to find a way to achieve both, because they feed off each other and are related.
It's as if I asked "do you want company growth, or customer happiness?" Try both!
> The reason playbooks have sprung up, is because 99% of use-cases for businesses are for the most part "solved." We're not in the era of renting colos and manually setting up MySQL anymore (which the author advocates for in order to return to "creatives," btw). We have managed solutions that you can throw money at.
I disagree they're solved! Again, this is in response to a 3-part series basically saying "everything is expensive and everyone is miserable and fewer businesses are succeeding." From a profitability perspective, when was the last Google or Facebook made?
> And good riddance to all that too. I don't need to work with any more software engineers who think it's okay to be a massive dick to everyone just because they can implement a CRUD app.
To be 1000% clear: I also hate(d) the machismo/showboating we got from the rockstar/ninjas era. I'm a former theatre artist and I'm mostly about embracing people and creativity.
I think the analogy is a good one but needs to be tweaked a bit.
"Rockstars" make tens/hundreds of millions or even billions. A good software engineer can make as much as a doctor (in the US) or accountant. You don't have to be a rockstar to be a 400k/year accountant.
The pay in tech has never been high enough to justify the sort of extreme talent competition in sports and hollywood. Even some of the greatest software engineers (e.g. Guido Van Rossum) are paid pretty modestly in the low seven figure range.
Real world Rockstars like Taylor Swift live a glamorous lifestyle, travel the world, and tend to work short, 2-3 hour gigs. Nobody with a rockstar personality is going to sit in a flourescently-lit office building staring at screen for 8 hours a day. No matter how many ping pong tables and free kombucha kegs there are, it is a very dull environment and does not attract the sort of ambition that rockstars have.
> Nobody with a rockstar personality is going to sit in a flourescently-lit office building staring at screen for 8 hours a day. ... it is a very dull environment and does not attract the sort of ambition that rockstars have.
Maybe I'm just old, but when I think of a rockstar I think of a lead singer or guitarist that has spent most of their life practicing their craft and loves it so much that when we see them perform it's just a taste of what's inside of their head and heart.
There's at least several tens of thousands in 'tech' worldwide taking home mid 7 figures to 8 figure salaries USD or USD equivalent, and not including those in the C-suite either.
They are just rarely mentioned on HN because there's already enough tech billionaires to exhaust the patience and attention span of most readers seeking out that kind of content.
If you want to be as profitable as Google or Facebook, you probably do need some rockstars, but you also need to be developing a technology or business that produces tremendous amounts of value. You can’t just hire rockstars to work on something low margin or without significant demand and expect because they’re amazing engineers that you’ll make a lot of money. Conversely Airbnb has made a lot of money and is just CRUD - they pay a lot and I’m sure there’s some fancy stuff behind the scenes, but you could probably copy most of the user visible functionality with a team of bootcamp devs.
I feel like the tail is wagging the dog a bit here because if what you’re actually trying to optimize for is building >$100b business, hiring rockstars or creating a rockstar culture is just one aspect of getting there which you may not even need.
> If you want to be as profitable as Google or Facebook, you probably do need some rockstars, but you also need to be developing a technology or business that produces tremendous amounts of value.
Definitely agree on this.
> You can’t just hire rockstars to work on something low margin or without significant demand and expect because they’re amazing engineers that you’ll make a lot of money.
This isn't what I'm trying to argue though — I think when you reach for the ceiling instead of avoid a floor, you're more likely to find those tremendous business value opportunities. Consider what Google shipped/built under Schmidt, "20% time," "Don't be evil," and "we're a different kind of company" vs… the last 10 years?
I'm not saying "being wild and creative makes a bad business good," I'm saying "embracing some narratives of a creative industry may increase output and be more cost-effective, even if playing "outside the playbook of the 0% era" is scary."
---
On another note, I'm regretting that "rockstars" is in the title . It was meant to support the idea that tech was more of a "creatives" industry (and the organic use of that to mean "great talent" is a symptom of that); but I think my "break playbook and embrace creativity" is being read as "let's go back to rockstars/ninjas! Genius hackers who don't sleep and don't care about HR!!!!" which is… not how I feel haha.
That’s fair, I think I strawmanned you a bit - I know you’re not arguing that taking a creative approach to software will automatically add value.
To your point about Google, that approach did lead to gmail, but also many abandoned projects like Buzz and Reader that led to Google’s infamous reputation for canceling products.
My original point remains: I think the business requirements and goals should inform the culture and hiring processes of a company. If you don’t need to deviate from the playbook to get things done, why should you? And not every developer wants to deviate from the playbook either, some prefer stable predictable work - while I’m not personally in that camp I think it’s not a moral failing, doing unpredictable things is more stressful as more things can go wrong, you might have to work more or suffer the consequences of delays/starting over, etc.
If you’re a bank, or a web consultancy, you want boring and predictable solutions. Conversely if you’re OpenAi you want people willing to reason from first principles and invent crazy new things because you’re shipping cutting edge technology at massive scale - and I think they’re doing that. So I guess I’m left wondering where exactly you think opportunities are being left on the table due to overly rote software development practices that would be improved by creativity?
The term "creative" has always bothered me as it seems to usually be applied to roles that don't actually create anything other than pretty pictures/designs/words. I guess it makes sense from the aspect of "creativity" but it seems like those who glom onto the appellation of "creative" almost always rely on someone else to do the heavy lifting of building (the actual creation of a product) and bringing anything to market.
> It's as if I asked "do you want company growth, or customer happiness?" Try both!
Obviously, both are "good," but again, what's the end goal? To truly be the problem solver, you must make the hard decisions and tradeoffs. Companies trade-off growth versus customer happiness all the time. Right now, a lot of the largest companies are the ones that prioritize growth.
To loop back to the point, obviously employee morale and a good end product is "good" and can have synergy, but again, what's the end goal? At some point, you must prioritize. A lot of companies have been very successful prioritizing the end product versus making their engineers feel good, which is why I think this topic recurs.
> I disagree they're solved! Again, this is in response to a 3-part series basically saying "everything is expensive and everyone is miserable and fewer businesses are succeeding."
I some ways, I think the 3-part series and your article are basically saying the opposite. The 3-part series acknowledges the market conditions and overabundance in the past, and says that in order to get the same results, we need to be smarter moving forward. Whereas your post is (in general) saying that it use to be really good in the past when we were treated like rockstars, we should return to that.
And returning to the point, technical problems being "solved" is not mutually exclusive with a general market downturn. That is to say, you can't point at the general market downturn to say that deploying a CRUD app is not a solved problem.
And this gets to my main point (which I think the author of the 3-part series would agree with): returning to the "rockstar" era will absolutely not bring software companies back to the crazy valuations and money. The "rockstar" era was the result and not the cause of the crazy valuations of the past.
> I some ways, I think the 3-part series and your article are basically saying the opposite. The 3-part series acknowledges the market conditions and overabundance in the past, and says that in order to get the same results, we need to be smarter moving forward. Whereas your post is (in general) saying that it use to be really good in the past when we were treated like rockstars, we should return to that.
I like this framing, thanks for it. Typically, when something is suboptimal, there's one argument that says "go back to before we [X]-ed" and another that says "no, we need to [X] smarter."
Illustrating this, [on another post of mine, I discuss how Alex Russell effectively says "SPAs/React are terrible" and Laurie Voss has a rebuttal of "no, we need better frameworks, smarter frameworks!"][1] Another example is when FB was in trouble of Cambridge Analytica stuff, it seemed like the answer to Bad Facebook was always More Facebook.
I'll look over it again, but I didn't get a giant message from Kellan's posts of "we need to be smarter moving forward," more about "how we got here." And I certainly don't remember how we're supposed to be "smarter moving forward."
But I also don't think Kellan and I disagree too hard on anything actually concrete. From [the footnote][2]: I think we both want technical decision-making to be grounded in business reality. I think a lot of scared leaders took the "Boring Technology" slogan to mean "something that doesn't scare me," which is IMO in practice was variations of "we won't innovate" / "we'll operate like money will always be free" / "we will use the most popular things I see in the blogosphere."
---
Said it in another comment, repeating here: I'm regretting that "rockstars" is in the title . It was meant to support the idea that tech was more of a "creatives" industry (because the organic use of "rockstars" to mean "great talent" is indicative of this); but I think my "break playbook and embrace creativity" is being read as "let's go back to rockstars/ninjas! Genius hackers who don't sleep and just HACKHACKHACK!!!!" which is not how I feel. What I'd like to go back to is taking some technical risk when there's a good justification for it, even if it's not what Uber did for their problems.
---
> That is to say, you can't point at the general market downturn to say that deploying a CRUD app is not a solved problem.
Well, many developers who were around in the Heroku era think it's harder to deploy a simple CRUD app than it felt like it was then :-p
But I'm also not talking about CRUD apps, I was talking about building a cost-efficient technology org for a better business. That's why the examples I used (deployment pipelines that costs tons of money and entire teams of developers to build and run) tended to be expensive. Many in the industry called these things "solved," but mostly ran giant bills. Your initial comment said "you could throw money at it," and I never thought that was the way, but especially now that money isn't free.
---
> And this gets to my main point (which I think the author of the 3-part series would agree with): returning to the "rockstar" era will absolutely not bring software companies back to the crazy valuations and money. The "rockstar" era was the result and not the cause of the crazy valuations of the past.
Again (and it's on me that this isn't more clear): I'm not saying "RETVRN TO ROCKSTAR," it's more "I think we should move the slider a little back to the "creatives" era." But while I don't know if it will create crazy valuations, I think it _can_ lead to happier developers, doing better work, and probably more profitable businesses, even if their market caps are lower.
It's interesting. I work for what is now a Big Tech company, but I started before the IPO.
We built a lot of in-house infra. We had few runbooks. We ran tons of high-scale services with a pretty low corresponding headcount. And we had a great culture, that was definitely more personality based than the rest of Big Tech. Individuals had Opinions (TM). On-call was about having a durable understanding of the system and sleuthing around to figure out what was happening (along with a strong emphasis on learning from failures and putting systems/practices into place to fix those.) And we, as engineers, were weird. My tech lead rolled into work at 11 AM every morning on their skateboard. I used to take a 2 hour daily break from 2-4 PM, go home at 6, and work again between 8-10 PM (ish, this was flexible) before I went to bed and was known to enjoy coffee with fried chicken. Our infrastructure and our services reflected our personality quirks and the interpersonal relationships we had in the office. On-call was one of my favorite things at the company because all the different personalities would weigh in with their different quirks and would offer different perspectives on big problems, and together we'd solve tough issues.
Somewhere along the ZIRP-age this all changed. We needed to become a Real Company (TM) now. We needed to have runbooks. We needed to have Boring Technology. We were poised for huge growth after all (aka our stock value shot through the roof.) The new managers we brought in needed to build out new teams to grow fast, fast, fast. How can we grow when we had such immature processes? We started building runbooks and hiring teams to manage them. Incidents started requiring 3-pages minimum of paperwork to document, which was enforced by an incident management team. Teams dreaded these incidents and where once we collaborated to fix problems, now teams became defensive and combative during incidents. Now we needed the incident management team to force engineers to cooperate during incidents because each team was trying to be as defensive as possible. Managers stopped thinking in terms of individuals with strengths and weaknesses and started thinking about headcount, both cost and allocation. Headcount became everything. With this change, attrition also spiked.
What used to take 1-2 engineers to spin up over a week or two now took months. We load tested our new services, but now we needed to make load tests repeatable and runnable in a runbook. Teams became extra defensive about features because the cost of every incident became so high. Nobody wanted to be the team that missed an integration test which came up as a cause of an incident. Our net reliability didn't actually change though the thinking was that repeatability would allow more seamless swapping of headcount. Program managers managed migrations and we started creating status meeting meetings which rolled up statuses of multiple ongoing initiatives cascading over multiple teams.
My own experience at the company went from having fun writing code and interacting with quirky folks at work to dealing with engineers who were dotting their is and crossing their ts in every aspect of their job. The managers treated us as headcount and so headcount we became. It's been a highly depressing arc to a job I loved and where I built a lot of high-scale code at, but perhaps the most frustrating has been watching our velocity decrease despite our headcount ballooning due to the overhead of programs, migrations, and incident management that developed their own bureaucracies. The saddest part is that the oldest parts of the company have become so essential and the bureaucracy so thick around them that replacing them has become really challenging. The parts of the company that were developed with the least care are the hardest to replace.
Great comment, I think a lot of us can find something in it that we relate to, even if not the whole story.
> Nobody wanted to be the team that missed an integration test which came up as a cause of an incident. Our net reliability didn't actually change though
This caught my eye because I’ve seen similar phenomena as well. No matter how well meaning those processes are very rarely do apps get significantly more reliable. It still seems entirely down to how complex the software is, it’s very hard to integration test your way to multiple 9s if it isn’t simple (some companies do it, but at bigger cost and slower velocity).
As for some other points…
I’ve come to believe change in culture is really something inevitable after a certain number though. And that number for an engineering team is not even hundreds, but may be only dozens. You operate less as a team and more like an army, with all the personnel issues that come with it.
What interests me lately is the idea that, for many companies, 80% of the value/growth seems to be delivered when the company is relatively small at X headcount. Then increasing the headcount to 5X does not deliver much more than incremental changes built on top of the core product.
Now if you have that many employees you need to find things for them to do, like managing themselves, creating processes or going off on new market bets, and you risk losing focus.
But at the same time a company with 5X headcount seems more attractive to investors than one with X. The measure of success being only an IPO means there isn’t much pride in being well-run, small and focused, admitting that you’ll never be the top player in the market. That precipitates changes in developer culture.
>Somewhere along the ZIRP-age this all changed. We needed to become a Real Company (TM) now. We needed to have runbooks. We needed to have Boring Technology. We were poised for huge growth after all (aka our stock value shot through the roof.) The new managers we brought in needed to build out new teams to grow fast, fast, fast.
I agree. We should have had a cleansing cycle during the financial meltdown and cleared off all the madness. Instead money was printed and the party had restarted with even more stupidity.
I believe the new "rockstar" developer is one who can completely subvert their own ego in order to better serve the business/customer/team.
The age of toiling away with weird tech in a dark corner and then expecting the team to be amazed at your contraptions is 2000% over. I've definitely been through this lesson a few times myself. It works and works until it doesn't and then one day you find that you are left holding a really big bag. My new ego trip is watching junior team members quickly ramp and become productive. Maybe you don't have to turn that ego off at all. Perhaps you just need to find a way to redirect it and get it hooked onto a new, more productive target.
If you want to go try crazy new stuff - do it on your own time. I really don't understand why this is controversial. It's the best of both worlds. I've heard some excuses for why this is infeasible but they're super-duper bullshit to my ears. If it works on your home lab and it is obviously a cool/valuable thing, then maybe put together a 5 minute pitch deck and present it to your manager/cto/etc.
Maybe for CRUD business apps this is true, but there are constant innovations and research papers coming from rockstars in dark corners of the world. All of the time.
I've seen teams today have just as large an FTE team dedicated to infrastructure as my employee 10 years ago to manage a set of cloud services that cost wayyyy more than the colo hardware we had doing basically the same amount of traffic. So people cost hasn't always gone down as infra cost has gone up. "Shitty internal heroku" has some advantages once you have engineers that have been at the company more than 3 months: there's just not nearly as much surface area to understand + you have the source code right there if you really have to get int here. Let's leave judgements out of this - you can just as easily build a fragile tower of cloud services and abstractions to "feel smart" as you can reinventing serving an application (reinventing is a strong word anyway... it's just a different set of off the shelf tools usually). And in many cases "managed" services manage the easy part (standing up the hardware and installing the service) and don't manage the hard part (configuring the hundreds of knobs to optimize your particular use case) particularly well.
I'm not close enough to the metal for those services/infra teams, though, to be able to completely tell if the FTE hours spent on all that cloud stuff are necessary. That is - can you throw stuff into the cloud and set-it-and-forget-it and not deal with ongoing cloud/infra/config maintenance? But one of the main things those teams seemed to often focus on was spend - if you leave it unattended is the cost just going to eat you up? But then there's a big irony.
My experience with cloud migrations was that you could migrate and forget about it. This was for a high traffic metro newspaper, that was easy to cache for external users but needed to handle a large newsroom using wordpress to manage the content (wordpress can be a huge resource hog).
The main costs in terms of labor end up being making sure that backends services are updated. Same pain you’d feel in a colo, except you cloud provider may force an upgrade you’d otherwise wish to put off. Now I intentionally kept most things at the VM level of abstraction, because it was clear that the added complexity of something like Kuberetes wasn’t worth any savings you might get by needing one less server, and I had enough granularity of healthchecks to just automatically spin down any server that was causing problem and let autoscaling work it’s magic. This was also 10 years ago, some choices might make less sense today, YMMV.
Also, don’t think I’m saying all those people aren’t necessary based on the current needs of the business. I just knew I was in a very constrained environment and prioritized anything that allowed for the constant shrinking headcount my department was facing.
Wasn't running them all, but a couple recent companies had ratios of about 1:4 and 1:2 (employee cost : cloud cost) with cloud spend in the low-to-high hundred Ks annually.
Curiously, the company with the higher cloud bill also had the lower ratio (spending more on both, but proportionally more on engineering); my diagnosis after the fact is that they were building for a "do 10x the scale" future that was never realized but which would've pushed the cloud spend higher without needing more dev spend. In the future, I wouldn't spend so much that far in advance of actually needing to scale.
I feel like the author themselves is a bit scattered on this point.
In the very quote you lead with, they link to another post[1], which says "...MySQL is boring. Postgres is boring." Our author says they don't like that Boring approach, but then they advocate for a "pets, not cattle" approach, which I think is in line with administering a MySQL server, or such.
Maybe this all comes down to different people's definitions of Boring. Maybe modern developers think of "Cloud everything" as the boring/playbook approach, while other people (like me) see the older (in my mind simpler) idea of administering a few machines, maybe even on bare metal, to be Boring.
You wrote:
> So, do you leverage those solutions to help you solve your problem? Or do you let your engineers re-invent the wheel so they can feel smart?
In my charitable interpretation of TFA, I think they are saying to let your engineers use smaller-scale solutions/technologies that address the problem at hand, as needed, rather than getting pulled into the Cloud-everything solve-all-your-future-problems-you-don't-even-have-yet ball of wax that's often advocated these days.
I can get behind that sentiment, but yeah I'm not a big fan of the way it's framed in the article, and I have never liked the "rockstar" term.
I think you're suffering from a lack of imagination.
> 99% of use-cases for businesses are for the most part "solved."
History is over, technology is done advancing, every business idea has already been had and executed well, everything that can be done has already been done. Sure. And you would have said the exact same thing last year, and 10 years ago, and 50 years ago, and 400 years ago, and you would have been monumentally wrong each time. But June 29, 2023 is different. This is the first day of the end of history.
By the way, the quotes around your word "solved" stand for many things, including paying through the nose for ineffective B.S. jobs that only exist due to inertia.
> do you let your engineers re-invent the wheel so they can feel smart?
The Curiosity Rover's wheels, which were re-invented for that purpose, are getting holes in them because the kind of rocks they expected to find are slightly different. Perseverance had its wheels re-invented again to fix this. Someone should have told those so-called rocket scientists at NASA that they're only trying to make themselves feel smart! If we never re-invented the wheel, we'd all be driving around on carved stone. "But you're not NASA, loser!" you say. Why not? Why aren't you doing something new too?
> the talent alone is the difference between millions of dollars
This is still true. One single talented coder can be the difference between a multi-million dollar business succeeding or failing. I have no idea why you think this is not true anymore; my guess is that you're completely unable to recognize these people when you're around them. Don't worry, most people can't recognize them.
In before: "wow, you really think you're an underappreciated rockstar, huh?" -- potentially. Of course I can't prove it over the internet. But rockstars are as much of the product of the environment you put them in as they are their own brains. Maybe rockstars actually are more common than you realize -- maybe you're the reason they're not rocking out.
> If you have a really talented coder who can be interchanged with another really talented coder, you do not have a "rockstar."
Or maybe the shackles and chains you're putting around these talented coders are forcing them into the same sub-optimal modes where they can't shine. I suspect if you really fostered an environment where software engineers could grow, learn, and shine -- where you fucking TRUST them, in other words -- you'd find they're not so replaceable as you think.
> I don't need to work with any more software engineers who think it's okay to be a massive dick to everyone
It's never okay to be a massive dick to everyone. In my experience, dickishness is negatively correlated with skill and talent, not positively. This is not an argument against highly talented programmers.
The reason why we don't have rockstar engineers is because the markets been filled with talentless hacks who can't code their way out of a paper bag without a jira epic breaking it down for them.
I had the pleasure of pair programming with ye olde rockstar for two weeks and we added more value in those two weeks to the company than the rest of the team did in a year.
Appreciate the candor. But we're all just here to get our fill, baby. B)
I only rockstar when my incentive structure has an unbounded ceiling. If you're paying me a salary, plus meager bonus, plus some bennies, plus some lottery tickets -- you're gonna get me at my 40-60%. Just enough to be above average.
Throw in some profit-sharing, and now we're talking, baby. What ever happened to that? Share some of the pie and I'll jump as high as you need me to.
Why is it foolish? I get the cynical attitude around not feeling like you're being taken advantage of; feeling like a fool. Is that the core reason? It's the sentiment (nothing wrong with that)
I ask because while I don't reject this position, isn't there something to be said for intrinsic vs extrinsic motivation: one's internal notion of excellence, pride, discipline, respect, mastery, craft, grit... insert various honoring-self words.
Seems precarious to me, to let external dictate ones concept of value. Btw, I suffer from "doing my best" so that's why I'm curious, I am not shilling for BigCo.
You bring up good points. And like all good points, they stand off to the side and proclaim nothing -- or atleast not enough that it's easy to refute them. So the only guiding light left is our values -- and what we personally believe.
I do not have notions of excellence, pride, discipline, respect, mastery, craft, grit, or anything of higher substance inside of me. At one point in my life, I'm sure I did, but they have been beaten out of me. Some have been taken advantage of by the ulterior motives of others; and some have simply not been conducive to achieving my goals (i.e. having fun, putting bread on the table, not ending up in the streets, living a leisurely and relaxed life, etc.).
In another vein, one could make the argument that the precarious thing is to tie one's intrinsic motivations -- those little devils inside of us -- onto a "job." That job will one day end, and you will have to find another as an effective outlet for those little devils; otherwise depression and all sorts of psychological illness will start encroaching. If one wishes to keep one's values strong, why make them dependent on uncontrollable things? Unless you are independently wealthy (if that is the case, I can offer nothing), there will be times you will be required to make sacrifice of your values to "get the job done." You will have to be sloppy, amateur, lazy, selfish, and maybe even villainous to do what needs to be done. At that point, you can hold strong and suffer the consequences, or you can relent, tell yourself you are temporarily setting aside higher ideals (it never is temporary), get the job done... and suffer the consequences.
Tying your soul, your essence to such a thing is precarious. Putting your heart into things that will require you to abdicate your integrity is foolish. You will end up hurt, broken, and missing pieces of yourself. It is better, and more sustainable, to save that sort of thing to matters that aren't directly related to your survival. Otherwise, to charge steadfastly into what one knows will be suffering is noble, but foolish.
This is a really helpful reply. Thank you. Some thoughts:
Excellence is not in contrast to fun, leisure, and relaxation; it's made that way via Capitalism and hustle culture.
> In another vein, one could make the argument that the precarious thing is to tie one's intrinsic motivations -- those little devils inside of us -- onto a "job."
Well said. I suppose it does take more energy, effort and soul really, to constantly decouple one's job from identity and worth. That one's internal values _ought_ to permeate through one's life "authentically" is certainly easier said than done.
Thanks again for responding in earnest. I'm convinced lol.
"Talentless hacks" is a bit too far, but it's very true that over the past decade, software development has been flooded with people who don't care about software development: people who never touched a desktop computer until university, where they only went into CS because they knew a $150k starting salary at a cushy job was waiting on the other side.
Nothing wrong with them doing this, of course. And they're very smart people. But you really can tell the difference between "software developer who takes pride in their work and wants to build something great" versus "software developer who wants to pay their bills". It's the difference between the developer who responds to a packaging problem by learning their build and dependency management system inside out to understand what's going on, versus the developer who copies and pastes from SO/ChatGPT until it works for some reason and moves on with their life.
Granted, not all software development companies deserve the former. But a tightly focused startup with 50 developers from the former category can absolutely dominate the market, at least until they get acquired, as we've seen so many times before: e.g. WhatsApp, Instagram, OpenAI.
Since when is the CTO expected to be to be a rockstar programmer? CTOs communicate technology matters to the board that they are beholden to. And they manage the technology teams within a small region towards the top of their departments pyramid.
These are different skills and responsibilities from being an elite rockstar developer.
CTO here. I spend little time with the board, and most of my time with customers and employees figuring out how to use tech to make life better (and make more money) for those customers and employees. I do a good job, the main interaction I have with the board is, "Good job, have more options and a raise... and the valuation of your shares has doubled." The other interaction: "Help us go raise some money." I do a bad job, and the interaction will be, "we need a new CTO".
Oh, and I still code. Because I like to code. A lot.
I found the tool submission you're talking about, I remember putting my username into it when it was first shown and it did produce a list of accounts with % probabilities (in my case all unrelated to me), but when I tried doing it again now it says "This user does not have any public activity in the previous three years." for "swores" (and I've multiple comments just this week).
Luckily for the throwaway person above, it says the same for them. But does still show results for some, maybe only big accounts now?
edit: ahh, I think it just took a one-time snapshot of the three years leading up to its 2021 creation, as my old username that I changed to swores does show up. So it's an interesting proof of concept that someone could do similar analysis, but not an ongoing tool. So it won't be telling us which CTO to avoid :P
Because I'd sink the company I'm golden hand cuffed to if I said what I think about the engineers I have to manage. But I do think I'll be bringing in SAP timesheets to share my pain with everyone:
> We need to keep track of the time your spending on your opex KIPs. We can't pay your expected bonus if you don't.
you're the CTO of a company mostly composed of talentless hack engineers that can't code their way out of a paper bag? You're literally the apex engineering lead; that's so... sad. Do something about it.
The budget I have for hiring is tiny and no one on the board believes 10x engineers exist. The choise is between not shipping anything at all or having to deal with said talentless hacks who happen to know more than me about tool X but otherwise slow development to a crawl.
Turns out reading HN comments is not a good way to run a company. Who would have thought?
I mean I also sure hope nobody is reading your comments to learn about running a company! They'd learn about "not answering the question," though, I guess.
Remind me, whose job is it to convince other execs and the board of the most appropriate talent strategy? Maybe lesson 1 here is "if you're running a company and don't trust your CTO, look for a new one"?
It sounds like you hired a personal mercenary, not a 10x engineer. Everybody is more productive when they ignore the rules everyone else is expected to follow.
There's probably nothing wrong with your existing team except it being too large and democratic. One guy can bounce ideas around in his head faster than a dozen negotiating them with each other. I've seen teams like yours, and IME the issue isn't always incompetence, sometimes each of them thinks they're the rockstar. Anytime one of them doesn't get their way, they jam everything up for everyone else.
We've become too democratic as a whole. Nobody follows orders from above; everybody expects to have a say in things they don't even have a stake in. Whenever you get to hiring again, try stacking your team with 2-4 ex-military types and see how it works out for you (it works well for UPS well beyond that scale). You don't get bleeding-edge experience (and never a 10x-er), but you get tax breaks, competence, obedience, people trained to work as a cohesive unit, and you can wave the flag and call yourself a patriot (which helps with government contracts if that's your business).
> Turns out reading HN comments is not a good way to run a company. Who would have thought?
Literally everyone thought this, and probably even the talentless hacks that you hired. Strange that you had to create a throwaway account to figure this one out.
"When WhatsApp was acquired by Facebook, they had 35 engineers. They were able to do this in part because they didn't "follow playbook," and while you probably can't go that extreme..."
Not my area of expertise at all but, 35 engineers for WhatsApp sounds like a lot
I think "rock star" is a superficial term which is mostly poking fun at some programmers for having long hair, liking rock music, maybe even owning a guitar.
I almost agree with this guy except when he suggest to use something like Java instead of say Python to be able to scale.
Most of the successful tech companies he is referencing are using Python. And no one saw a big successful website/app in java. Most of the time corporate or gouvernement shitty web applications that randomly crash with backtraces when you are using them.
Maybe it's just me but I hate the word "creative". I don't like that engineers are refered to by the same word that describes actors, poets etc. Problem solving is subposed to be rational/unsurprising, most of the time you follow a logical flow and a consistent methodology to come up with solutions that make the most sense. "Innovation" is supposed to be a side effect of that.
I took the "boring tech" stuff differently. I choose technology that makes my production environment boring.
Sometimes that means picking technology that is not mainstream. Choosing mainstream technology is one of the best ways to make your production environment decidedly not boring.
I would argue Erlang is one of the most boring pieces of tech you can pick. Look at how boring your production systems can get when you choose that.
> Getting flown as an undergrad to SF in 2009 for an interview at Facebook, put in a hotel, served free food, all because you could code? You felt cool
Sorry, you drank too much of the kool aid. Let me clear it up for you: you're a cost center that the corp would rather not have.
I kinda thought coding might be immune, because it's about information. But of course, you build on lower levels. Today, you can get outsized returns by applying LLMs to real problems. Also about information.
There's threads of a few interesting ideas in here, unfortunately the presentation teed them up for the current cultural mythos to obliterate them.
A number of years ago, the uptight cultural conformity of the valley bubble branded itself with some of the imagery the author attempts to borrow. The collective consciousness hasn't forgotten how that image was power-blasted away. It was done as cover to sneak in a bunch of orthogonal changes while nobody was looking, but that part's neither here nor there. Forget I even mentioned it. The end result is that we're here now and we have the tools to evaporate this article with one shot at 50 paces.
This is written in the style of someone who worships software development, but is not themselves a professional developer and therefore has spent zero time working on a team to build something more complex than, say, a webpage.
I recognize the author is a professional developer, and seemingly has worked on teams to build complex things, but it appears as if they forgot how the human machine works, and are now stuck on some kind of nostalgia trip where everyone involved in the process is somehow like them and only able to gain energy through self expression in their software engineering work.
I do wonder sometimes if there's something about living in SF or NY that makes you forget how tiny your perspective actually is.
Also the reference to Taylor Swift. I had to re-read that paragraph to see if it was a good thing or a bad thing. Could've used, you know, Eminem or M.O.P or something.
Almost the same specs, but the one in the startup had better scalability and fewer serious bugs...
So, spend 10-50x less and get higher quality?
Worst is I now consider case b) quite lean and efficient compared to what I see tech consultancies doing towards public sector, banks, etc...
-- My point being: There definitely IS a room for "rockstars" (don't like that term, but interpret this as developers competent enough to carry a lot of weight on their own and work in a smaller company) to deliver a lot of value.
The problem with the model is the lack of predictability (what if the few developers you trust don't deliver), the bus factor if one of the few devs quits, etc
In a sense inefficiency is a goal, because otherwise you loose redundancy. It costs a lot to hire 100 to produce the output of 10, but much lower risk.