The thing about Scrum is the observations and principles make sense, but then to sell it as a course they've turned it into very specific prescriptions.
I went on a scrum course and the takeaway was basically that feedback is a big deal, and you should try to get some repeatedly and quickly. It's also common sense that you should have tasks written down somewhere, and that some of them are more important than others.
You certainly need to think about how long tasks will take, but there's no reason why you need to do planning poker, that just seems to be one among many ways to think about how long something might take. Tracking velocity is another one of these things that seems replaceable.
If you have a team of people that more or less adheres to a few principles, there's no reason you can't get things done.
The problem with any system is that people try to enforce it, military style. In my previous job, we did scrum, but not too strict. We had two week cycles, not-too-strict deadlines (most of the time) etc. If I finished my task early, I was free to pick up tasks from the planned list, without having to get permission from my manager. We also didn't agonize over story points, retrospective etc. We did it light hearted and quick. The only thing we religiously followed was daily, quick 15 min stand-ups. Everything else was flexible.
My current job is the opposite. Some tasks take 15 minutes of discussion (no, they aren't complex tasks) with people debating whether it is worth 5 points or 8. It is just tiring and pointless. And retro - gawd, I hate those. There are all kinds of stupid shit (people using references from music, movies etc, trying to make it "fun" and "hip"). I can't bring other tasks into the sprint, even if I finished all my current tasks, without my manager's permission. And on and on.
I was thinking the other day why this is so painful and awkward. I realized that it all comes down to "metrics" - end of every sprint, my manager has to present it to his bosses, with pretty graphs describing "velocity" and other buzzwords. So he has to do all kinds of jugglery to appear competent to his bosses and not alienate the team at the same time.
All of this could be avoided by treating the team as adults, instead of trying to "processify" or quantify everything.
However hard we try, human beings can't be slotted neatly into buckets nor can they be precisely quantified. However hard we try, estimating software projects will never be an exact science.
A cause of this problem is when people are worried about what happens when they say "erm, I don't know how to do this cleanly" and so they want the "how" to be defined far far in advance. Solving this requires a mix of
A. Identifying those activities that will genuinely make people's lives easier if they have a process and designing those processes to be meaningful and low-burden. Those activities will vary based on team members individual peeves and affinities.
B. Creating clear lines of communication for how and whom to ask for help. This often means more clear naming of responsibilities.
C. Ensuring there is enough slack time for people to be able to help each other out.
D. Creating trust in the team that asking for help won't lead people to question if you are fit to do your job.
" I realized that it all comes down to "metrics" - end of every sprint, my manager has to present it to his bosses". I wonder how widespread this is.. Certainly that's exactly what I experienced at a previous workplace, and worse than that, people's performance was judged on whether they'd done exactly the "right" stories in a 2 week sprint. Rather than thinking about developing software, people were working out how to game the system to ensure their metrics were good. I cannot see how this can have done the employer any good. I've now moved jobs, to a place that's far more Kanban, and if mid-story we encounter bugs that need fixing, or opportunities to improve something, or something else useful that piques our fancy, then as long as the whole team agree and our overarching goals are being worked toward , we can work naturally, rather than rigidly.
The purpose of metrics is to be something to game.
Now you do need don't to anything illegal policies in place, and you might have a [unwritten] moral code of what you can't do. However the purpose is to game the metrics. For all the discussion that follows I'm going to assume we are within these limits.
If the boss wants profit margin as a metric, then you need to figure out how to do things cheaper, and how to get more sales, or higher value sales or some such, and sometimes get something off the books. Note that this last is why smart investors always ask about the bottom line - because the rules of what you can hide from the bottom line are strict enough that the metrics cannot be gamed to your disadvantage. That said off the books is sometimes a useful thing to have - sometimes you know you need to do something hard and so you need to leave an approved way to hide specific things in plain sight.
The problem is most developer metrics are things where gaming the metric isn't good for the company.
As a manager, we have been incentivized to game this system exactly as you describe, both from my end and my devs end. Moving to a company which only cares about delivered software instead of metrics saved my soul.
Interesting, I had a very similar experience in the past.
I think we just were part of a software factory that is not necessarily bad, it's just a different approach to work. It's much more convenient for a manager to just apply "standard" principles (scrum, 2 weeks, story points, velocity, etc. like a mantra) rather than innovating at this risk of being wrong, difficult, etc.
Too much bureaucracy is not good, maybe that was the purpose of this article. The more you have it, the more difficult it becomes to get out of it, to innovate, etc.
Agreed on the kind of religious adherence which can make the process feels a bit like..a staged show? Stifling instead of open discussion?
"Metrics" though drive when something might be delivered which triggers marketing spend, hiring requirements and so on. It says "we're working on this now and that next" which drives all kinds of internal storytelling about priorities and cascades down (or up?) into quarterly calls.
How do you do it otherwise without having ipod summers?
You set up long term team based goals (OKR's, Objectives and Key Results) and then evaluate how well those goals were achieved. It isn't 100% accurate or fair, but neither are Scrum burndown charts.
> whether they'd done exactly the "right" stories in a 2 week sprint
Not only that, but if anything else comes up during the "sprint", you're expected to address it while still finishing everything that you (involuntarily) "committed" to during sprint planning.
WE committed. We made these commitments as a team. If you can't fulfill what has been committed to by the team, then we as a team have to have a discussion about that. I'll put the meeting for next Monday on everyone's outlook.
Estimates are not commitments. I know young devs are preyed upon, but people can't be pushovers in the workplace. It's really detrimental to all, so we need to support reason above rituals and rigid processes.
Can you link me or someone else link me what you would consider the "ultimate kanban guide"?
I work in an environment that had no project management tools or methods being used other than emails and I've finally gotten buy in on Kanban and I want to make sure I guide my team properly.
Kanban in Action by Marcus Hammarberg and Joakim Sundén is a good introduction. In this book, the guiding principles of Kanban are presented almost like parables as the authors introduce you to their fictional team "Kabaneros".
Although I think he ended up with something more Scrum-like later at Spotify.
He also has lots of good videos about broader team organization (Tribes etc.) that I found useful for conveying some of these ideas to my non-technical peers.
> I realized that it all comes down to "metrics" - end of every sprint, my manager has to present it to his bosses, with pretty graphs describing "velocity" and other buzzwords.
You've discovered the secret of Agile in the corporate workplace: that the key takeaways, as far as the enterprise is concerned, are not finding better ways to develop software and better customer-developer relations. It's all about trackability, measurement, and metrics, because everything is. Trackability and measurement enable key decision makers to make accurate budgeting and execution plans and there is literally nothing else for key decision makers. Therefore, it makes less of a difference what you output than that it was properly planned for, tracked, and measured and that it hits organizational KPIs. That's why nobody cares that Scrum is a giant productivity suck. Code could be written faster, but that's not writing it properly.
Agile in the enterprise is a game of Mornington Crescent. The goal is not to foster the things advocated in the Agile Manifesto. It's to make it look like the company is fostering those things, while actually promoting the same Taylorist values corporations have always loved (and workers hated).
"Interspersed with the turns is h̶u̶m̶o̶r̶o̶u̶s̶ 𝗯𝗼𝗿𝗶𝗻𝗴 discussion among the panelists and host regarding the rules and legality of each move, as well as the strategy the panelists are using. The actual aim of the g̶a̶m̶e̶ 𝗰𝗲𝗿𝗲𝗺𝗼𝗻𝘆 is to e̶n̶t̶e̶r̶t̶a̶i̶n̶ 𝘄𝗮𝘀𝘁𝗲 𝘁𝗶𝗺𝗲 𝗮𝗻𝗱 𝗺𝗶𝗰𝗿𝗼𝗺𝗮𝗻𝗮𝗴𝗲 the other participants and listeners with a̶m̶u̶s̶i̶n̶g̶ 𝗿𝗲𝗽𝗲𝘁𝗶𝘁𝗶𝘃𝗲 discussion of the fictional rules and strategies."
Funny, retros are the one thing I like. If you can get a team to honestly say what went well and what went bad, and the step that can be taken to improve, every two weeks, with some light follow up. In 6 - 12 months, you can turn a flaming wreck of a project into something that resembles best practice.
I have do it as a consultant. I have taken a project on life support that they were gonna cancel, regardless of our performance, and resurrected it so hard they actually decided to keep it. We changed their minds
The most effective tool was retros. I dislike agile in all other aspects, and do not use story points. Retros are the best bit! It's the core part of the learning loop.
I learnt retros from Google. Big tech does retros. Its honestly magic when done well.
Retros are great iff there is team empowered to actually meaningfully change stuff. I've been in places where everyone agrees that something sucks but there is no actual way to change it. Then retros turn into an impotent bitching session... no thanks.
I'm baffled by this behavior. If you know it is the next most important thing, why do you care? What would it change in your behavior if it is a 5 vs and 8? If you are looking at a task that you would choose not to do if it were an 8 but choose to do if it were a 5, then you probably look for something that is more important to work on.
I think you are right that much of that is driven by managers looking for metrics, but unless they understand what the metrics mean it is pointless.
I think there's value in that debate (within reason) even as a developer. If Alice thinks a task is worth 5, and Bob thinks it's worth 8, then there's a good chance one of them knows something the other does not. Is Bob aware of some hidden complexity that bumps it up 3 points? Or is Alice familiar with a convenient library that solves exactly that complexity? Planning poker is a convenient time to get that knowledge out into the open.
Again, this is all within reason, and if it takes 15 minutes on every ticket, the team should probably work on their communication skills.
Everyone is correct in this thread. The stimulation of discussion is useful, and numbers are arbitrary.
Instead of worrying about 5 vs 8 though, the team should be discussing relative difficulty: is story B easier/more difficult/complex than story A? And then ranking stories based on the relative difficulty of each other.
Story points can then be derived based on that ranking, if the team chooses. They're useful for being applied to velocity calculation, and also helpful when picking up a story to work on (maybe a bad idea to start an 8 pointer on a Friday).
(I have a SM cert, working as a TPM. I applying Agile principles to teams, but modified to how they want to work and be most effective. No militant Scrum here.)
That for sure seems useful. "Hey Sally, you think B is harder than A. Why do you think it's hard?" "Hey Bob, you disagreed with Sally and think B is easier, why is that?"...is very likely to lead to a useful conversation in terms of everyone getting in sync and may well lead to hidden information being brought out into the open, which is a good thing. In general I think relative value/preference type conversations tend to reveal a lot.
Attention! Discussing story points is no longer allowed during planning. After voting, teams are to discuss relative story points, swapping points between each two story. This is expected to increase efficiency and coherence. Rule enforcement commencing beginning of next month after the first weekend.
My experience is that in well-running teams, devs are usually pretty aligned what these numbers mean. A difference in proposed points for some task does then usually mean there's a discrepancy in knowledge about it.
As for its usefulness: You of course don't have to necessarily poker to get these discrepancies, but it's a pretty effective way of going about it, I'd say.
Exactly this. It's probably not an accident that one of those numbers is half of 10 (and the number of fingers on one hand) – a common made-up number for everybody – and the other is 2^3 – a common made-up number for programmers.
I've always done planning poker with the Fibonacci sequence - 1, 2, 3, 5, 8. The idea being that the more complicated the task, the harder it is to estimate accurately.
Why is story point estimation tied to fibonacci sequence? Two generations of managers at my previous employment thought this way. It just seems so arbitrary to me.
First, the more difficult a task is, the more inherent difficulty there will be in "accurately" estimating the difficulty of the task. Fibonacci is used to represent the inherent lack of accuracy in more difficult tasks, since the numbers get _very_ far from each other as they go up the scale.
Second, the numbers _are_ arbitrary. Completely, 100% arbitrary. It's a _relative_ difficulty scale. Say you've got 3 tasks - A, B, and C. A and B are approximately as hard as each other, they're 1 story point. C is more difficult than either one - it gets 2 points. That's it. Story points are not, and should not be used as, a unit of measurement. The biggest utility is to identify big, scary tasks with lots of unknown factors.
The fact that they are _numbers_ is what tricks so many teams/PMs/management/etc into thinking that story points are more meaningful than they were ever supposed to be. Incidentally, this is also why some planning poker teams use t-shirt sizing (S,M,L,XL,XXL, etc). No numbers means people are less tempted to punch them into a spreadsheet while deluding themselves into believing that showing numbers going down is the same thing as "showing progress".
The closer the numbers are together the more time is wasted trying to be exact about them. Fib helps reduce the amount of back and forth. If you need to guess how much something weighs and your options are 1,2,3,4,5,6,7,8,9,10,11,12,13,14.....100, you are going to have a lot harder time achieving consensus than if you asks "Is it heavier or lighter than a breadbox?"
My team doesn't really argue about points next to each other (on the Fibonacci scale) anymore, we just pick by majority and move on.
The conversation is meaningful when it's about 3 vs 8 because exactly as you say, there probably is some hidden complexity not everyone knows about, or sometimes some work has already been done or there's a framework feature that makes it easier than it seems.
But 2 vs 3 or 5 vs 8... this is explicitly designed to be an imprecise number, so let's not waste our time debating which one to choose.
My team has a rule that if your Fibonacci point estimates are within one of a given unit, you just accept that estimate with no discussion - so if I think a story is a 3, and others think 5 and 8, we’ll take the 5 and move on. I think it gives a good balance of discouraging hair splitting and surfacing cases where having more discussion is actually valuable.
If you will do something different with a 5 than with an 8 then yes. If not, then the ambiguities you describe will be worked out when you do the work. It is just a vanity metrics if the results doesn't change what you work on next. It seems important but it has no actual bearing on the teams sequencing of the work.
If Bob had a plan for doing the work that didn't involve Alice's library, there's a good chance he would have just jumped straight in to implementation without talking to anyone, and maybe the library wouldn't have come up until code review (if at all). By identifying the complexity mismatch ahead of time, they saved 3 points worth of effort. This doesn't affect sequencing at all, but still seems valuable to me.
Or Alice is more familiar with that section of the code and would move more quickly in there and Bob would be doing more learning along the way.
Quick discussion of the differences is useful, but 15 minutes is ridiculous. Just take the higher value and move on. Eventually you’ll baseline at slightly more points per sprint on average, but they are imaginary numbers anyway and not really comparable across teams.
My last job, we each had a minimum required individual velocity in our weekly sprints. That velocity score was the same if you were a junior or a senior. So in that case convincing the team it was an 8 could mean the difference between keeping your job or not. It was the most soul destroying, stressful job I have ever had. The worst was the Rockstar programmers not seeming to grasp that by reducing the points total on a hard ticket they were dooming their teammates. It lead to everything you can imagine, tickets implemented as fast as possible so they met the letter of the ticket but crashed on anything not defined as the team raced to meet their velocity. Massive unpaid overtime and burnout. Constant crashes in production. Races to claim the 'easy' tickets, and sandbagging. Eventually culminating in 8 of 12 devs quitting in an 8 month window including all seniors. My new job is so much better.
Sometimes importance is related to size. Points are arbitrary, but lets assume that on average a developer will work the entire sprint on an 8 point task. The PM might decide that feature is not worth the time investment in that sprint, if they could instead get 2 smaller features out the door, or some bug fixed - they might prefer to deliver that to stakeholders.
That's why there's even a velocity expressed in points, like the tasks - it's the PM's budget to work with. So 5 points or 8 doesn't change the work you have to do as an engineer, but it changes the number of things a PM will put on the team's plate in the next sprint, and the points force them to make trade-offs, otherwise they'll insist that the bug, the small feature and the large feature are all high priority and have to be delivered in the next sprint.
I'm not justifying the 15 minute time investment for sizing, that is clearly way too much time, I'm only highlighting that in scrum, as an engineer, you don't necessarily know or decide what the next most important thing to work on is. That's the PM's job.
Bugs get resolved desk-side five minutes after being reported and pushed through on PRs that have 89% test coverage (still green!). Developer is a cowboy hero! The good thing is: you always get more bugs like this so everyone gets a chance to be a hero.
My last "team" was constantly paralyzed by this. The scrum master would go around and everyone had to vote on the number of points they thought a story was, which then led to extended debate. Of course, it can always be worse...near the end, they tried some idiotic thing called "planning poker".
1. Sometimes your team needs to give an estimate of when it will deliver a feature. The idea of pointing in Scrum is that you establish a velocity, which lets you get a feel for how long it's going to take to deliver a given piece of work.
2. Sometimes your team needs to make trade-offs, and often the right lens here is ROI. You can't estimate ROI if you don't estimate the I. (Although I believe agile tends to somewhat overrate the usefulness of the ROI lens.)
3. In the absence of 1 or 2, the act of pointing can help to catch the case where you miss a piece of complexity; if someone thinks it's 8 and everyone else is a 5, then maybe they are aware of some gnarly code that everyone else has forgotten?
I'm not arguing in favor of discussing each story to death, just making a more general justification for the practice. If you never need to do 1 or 2, then maybe pointing isn't going to be useful for your team at this time -- and that's OK too.
I'm sure in some cases this is just driven by managers needing metrics to measure their teams by, but the above is an attempt to suggest some reasons that the teams themselves might find the practice useful.
Metrics are not bad. Story points and velocity are usually bad metrics though.
It's not a coincidence the consultancy sector uses scrum. They get paid for their output, usually by the hour, and scrum measures output.
If a consultant implements a load of useless features that make a product worse, but do so very quickly, then that's a great success for them. It's not fundamentally different from rewarding developers for the number of lines of code they write.
Oh god, this perfectly describes the company I just quit. They went from using Trello, allowing us to choose what to work on, loosely setting story points and ... that was it.
Then they decided they needed the metrics on everything, so they switched to JIRA, started doing retros, setting strict points on tasks(reprimanded in retros if you messed up), using burndown charts to reprimand even more, and giving the product manager the power to dictate what I work on and in what order.
It went from being a great company to work at, to a company I ran away from. I have half a mind to send this thread over to them.
> My current job is the opposite. Some tasks take 15 minutes of discussion (no, they aren't complex tasks) with people debating whether it is worth 5 points or 8. It is just tiring and pointless.
That was my last gig. The whole organization eventually just collapsed under the increasing load of being "Agile".
At my work, when one of us gets too into the details of pointing, estimation, or process we usually realize, stop, and say "sorry--I was being a Scrumbag"
> I realized that it all comes down to "metrics" - end of every sprint, my manager has to present it to his bosses, with pretty graphs describing "velocity" and other buzzwords.
This is a thing that's pretty key to Scrum: velocity can't be used to compare teams. It is easy to misuse though. I would start every meeting I were in with "these points measures cannot be used to measure performance between teams."
I find "velocity" to be almost completely useless, both in principle and in practice. It's not only useless for comparing teams, it's useless for comparing the same team over time if anything that affects "velocity" changes: the team composition, the length of the iterations, the nature of the tasks being worked on, etc. Now, how often have any of us worked on a project where at least one of those things didn't change, and sometimes fairly frequently?
I mean, just the difference in an iteration with a couple of people on PTO, or a holiday or two intruding into it, will throw the numbers off. Now factor in all of the other fuzziness that's inherent in the process... yeah, no. Don't bother trying to calculate or track "velocity". You'd probably get better results from Tarot cards or animal entrails.
I too find velocity to be one of the dumbest aspects of Agile. It takes the stupid of story points and doubles down on uselessness and arbitrariness.
Velocity in physical terms is an instantaneous measure. It's the same in Agile, you've done X things in Y time. But that ignores the context of those X tasks and the context of Y time.
The X+1 task might turn out more difficult or have some other challenge that makes it take longer. Time keeps passing so at time Y+2 the velocity measurement is lower than before. Now all the PMs and other team members jump in demanding answers which now makes the task take even longer.
As you mention, some span of time can see team members out or unavailable. So the velocity is "low" but rarely is that context captured or bubbled up the management chain. So your team has "low velocity" which ends up generating worthless meetings, bad reviews, or all sorts of problems.
I've seen "story points" be totally useless over time on teams where "number of tickets delivered per week" was remarkably stable. But people don't like using that for some reason.
If you do that you can actually very easily switch to Kanban methinks. In Kanban you sort of have to size all work to be about (!) the same size so that you can make accurate enough predictions for your lead time and cycle time. That only really works if all work items are about the same size though, otherwise you get too many outliers from your statistics.
I've found that most Product Owners really care about predictability and less about actual estimates. They care that if you say you're gonna finish something by the end of the sprint, much more often than not, you will. Estimates and velocity and such things are just means to the end of trying to figure out which items are reasonable to take into a sprint together.
If your Product person knows that they can throw 10 "random" (but currently most wanted by stakeholders) work items into a "sprint" and 9 times out of 10 you will actually finish all of those, they're probably gonna be very very happy as they won't have to explain delays over and over again on their end.
> I've seen "story points" be totally useless over time on teams where "number of tickets delivered per week" was remarkably stable. But people don't like using that for some reason.
I don't see how this can be the case. If each ticket is estimated to be about the same size, and it sounds as though they should be, then if you're doing say 5 tickets a week, 10 a sprint, and they're all about the same size then each ticket you could say is 5 points, and you're doing 50 points a sprint.
That's probably pointless (hah) if they are all the same size; ticket sizing is more for if you have tasks of different sizes and you're trying to roughly combine them into something you can predictably deliver.
But if you're at a stage when your delivery is already this predictable, you may not need to use story points particularly. If it's mandated for some silly reason then you could use the above method to bat it away quickly!
We have sprint retros and generally find them valuable. I’m curious how one would even go about incorporating music and movies into a sprint retro? That does sound painful.
It's amazing how infantilized this industry has become during my short 10 year career. I'm so thankful I switched to product and don't have to deal with this BS anymore.
As a former engineer who used to deal with the bone-crushing tedium of fundamentalist Scrum, I have enough empathy for my team to not force this crap on them. I keep the process light, communicate a lot, and treat them like adults.
> The only thing we religiously followed was daily, quick 15 min stand-ups.
One of the worst parts of scrum in my opinion. Even if it's a "quick" 15 minutes (which in my experience it rarely ever is), the forced context switch ends up wasting more than 15 minutes of productive time every day.
> In my previous job, we did scrum, but not too strict. We had two week cycles, not-too-strict deadlines (most of the time) etc. If I finished my task early, I was free to pick up tasks from the planned list, without having to get permission from my manager. We also didn't agonize over story points, retrospective etc. We did it light hearted and quick. The only thing we religiously followed was daily, quick 15 min stand-ups. Everything else was flexible.
This is exactly how I've seen it run with success at several different companies (including how I run it).
In my mind, the most important value of Scrum is having a cadence to give people reasonable timeframes to think in AND ensure that a variety of different conversations happen frequently enough (discovery, planning, prioritization, execution).
----
> And retro - gawd, I hate those.
It sounds like you hate these because you aren't actually empowered to make meaningful change within your team.
We've found retros are one of our most important rituals. They help us identify risks, process, and team issues; often while they're still small. We quickly work those in.
When I ran Engineering for a startup, this is roughly how we did things as well. If a disagreement around pointing a story happened, we resolved it within seconds and usually just went with the larger size. It didn't really matter so much, since we weren't using velocity as a goal or something reported on a weekly basis. I did use it to roughly ballpark how many sprints it might take to deliver something, but that was as much story point math as I ever did and never used it to guarantee a date. I did have to fight a few battles with my CEO and other VPs around that, but, in the end, the results of mostly consistent and predicable progress with happier dev teams made those arguments moot.
We also found retrospectives to be pretty useful and each team was encouraged to find what worked for them. The result was that we had 5 teams that had some agile/scrum process, but created mechanics that worked well for them.
> I did use it to roughly ballpark how many sprints it might take to deliver something, but that was as much story point math as I ever did and never used it to guarantee a date.
This is basically how we use it. Typically, if we have a multi-sprint effort, we'll guarantee delivery in the last or second to last sprint. Anything before that and we're having conversations around scope and trade-offs we need to make to hit targets.
I've had several engineers tell me their 1.5x to 2.0x as effective as their prior roles while having better WLB.
>Some tasks take 15 minutes of discussion (no, they aren't complex tasks) with people debating whether it is worth 5 points or 8. It is just tiring and pointless.
If you're not personally responsible for deadlines on the project, it could seem pointless. But the difference might mean pushing the commitment for a feature out two weeks, which in a commercial project could be a big deal. Planning is hard: you've got to try to estimate something with incomplete information, and then reconcile differing opinions. But it's definitely not pointless.
Velocity is probably one of the most misunderstood aspects of scrum. It's a key metric for long-term planning, but it's not intended to be manageable. It's also unique to the team, and not intended to be compared between teams. Many managers are not used to a metric that they don't manage, and that causes a lot (maybe most) of the bad experiences people have with poorly practiced scrum.
A lot of places are looking for new people (even remotely), don't stay in a place that makes you miserable. You don't have to suffer for other peoples mismanagement.
The whole debating around agile, scrum, kanban etc etc, with seniors hating Scrum yet it being pushed ever and ever again made much more sense when I heard some guy talk about Shu-Ha-Ri (https://martinfowler.com/bliki/ShuHaRi.html).
Having a strict framework à la Scrum is very helpful for new developpers or new teams, where they don't yet have their marks and need to get a feel of what agility feels like. Being explained principles is not enough to know how to apply them: Scrum is not a bad starting point to be in the right-ish track without really knowing what you're doing. As you gain experience, it's natural to want to shake off constraints, that's Ha and Ri
Maybe if you don't fill a team with nine juniors that have no mentors, you don't get this problem?
It's not like FAANG isn't hiring boatloads of new grads every year. Why don't those people need scrum to get on the right track, but half the rest of the industry seems to?
Those companies are very engineering-led. Boatloads of excellent engineers; deadlines are "it'll be ready when it's ready"; marketing is (and thus marketing deadlines are) light, so there's less planning needed; featuresets are largely defined by the engineering teams.
Anything that is reasonably unpredictable in terms of requirements but needs to be reasonably predictable in terms of feature delivery speed is a good candidate.
We use it because we need to release versions of our software a chunk at a time, and it's the closest we can get to continuous delivery given that constraint.
I don't even know if it's a thing with new developers or old developers; I think sometimes you just get people who don't want to play along. Half-hearted participation is as good as no participation. You end up with user stories that are poorly written, poorly estimated, and you can't really reap a lot of the core benefits of Scrum like velocity forecasting. What you end up with is I Can't Believe It's Not Scrum; a shoddy clone of Scrum that never comes near the mark. You're forced to keep going along this way because not everyone agrees that you're failing.
Maybe people are tired of the nonsense that systems like Scrum bring with them? Even the name is a bit silly. And when you start naming roles and inundating people with rituals the eyerolls really get going. Why add abstraction to common sense?
Or maybe Scrum is really just an attempt to turn bad team members into good ones?
Good team members...
- Provide good estimates
- Cooperate to create clarity around requirements
- Work to divide big problems into small chunks
- Keep good track of their work in a shared format
Bad team members shun all of this and expect someone else to do it.
It also fits very well with feature factory shops: a story fits nicely into a single feature. God help you when you are trying to build a system from scratch with scrum overhead. Not everything is a story or a ticket. It's painful.
This one hurt, it's exactly what I spent the past 18 months doing - building a new system. Luckily my director was sympathetic but the Company loves Scrum and Metrics. Tried to change process unsuccessfully, so I quit to work somewhere saner.
Yeah it's awful, the amount of overhead each ticket generated. Testers want to test each ticket. The more I hear about Microsoft's old way of building software the more it makes sense: have a soec that can change.
Scrum is a specific implementation of some vague overarching concepts. Of course it's going to be prescriptive, that's the point. Else you're just doing "agile".
When you bring a prescriptive implementation, you can then leverage learnings from many orgs over many many years to handle all the edge cases instead of reinventing the wheel.
If you don't, or do "Scrum but not quite", then when things don't quite work out, you're on your own.
I'm no fan of Scrum, and rather use almost anything else, but that doesn't change that there's significant value in using systems that are mostly figured out and "work". Most of Scrum's bad rep comes from people who think they know better, tweak it to suit their needs, then realize they made a ton of holes in the system and their Frankenscrum doesn't really work. At that point you're better off just doing your own things. Which is what folks do, they abandon Scrum and do agile their way.
If you do "Scrum, exactly", then when things don't quite work out, being "on your own" is the best case scenario. Worst case is that someone yells at you and blames you because Scrum Objectively Works so it must be your fault.
I see no reason to believe that Scrum is a tightly tuned machine that must be precisely run to work at all, and any deviation from it makes it fall apart. In fact, if that is the case for Scrum I'd consider it a fatal objection. I don't believe the effectiveness landscape for project management techniques is shaped like that, I don't believe that the effectiveness landscape is a black pit everywhere just around Scrum only for a sudden spike of effectiveness to exist precisely where Scrum is located, and if it were shaped like that, I don't believe in the capability of anyone to have found and proved that spike. Since the only practical algorithm to explore this landscape is gradient descent, anyone carefully and thoughtfully exploring this landscape should never have found Scrum.
Or, to put it in less mathematical language, the idea that Scrum is super effective but if you deviate from it just a bit it all falls apart is complete garbage.
Do real agile, all the time. Scrum's only useful for raiding for ideas, and I'm not particularly impressed with it for that purpose, either. But it's not quite literally bereft of good ideas. I wouldn't give its ideas any particular credence for coming from "Scrum!", through. I don't see a lot of evidence that it is specially deserving of respect. It seems to work for some people, yes, I don't deny that, but the rate at which it works for people is sufficiently low that I don't think it has any sort of special claim.
> Since the only practical algorithm to explore this landscape is gradient descent
That's why an algorithm was not used. Effectiveness has not been sufficiently analysed to be reduceable to a line on a graph.
> Do real agile, all the time.
People who call something "real" without defining what "real" is may be well-practised at sounding nice and emphatic in meetings, but without any detail it's hard to see it as anything more.
The brilliant thing about Scrum marketing is it's pitched and talked about as infallible. If it didn't work, it's because you didn't do it right. If Scrum worked for you, you must've done it right.
> If you don't, or do "Scrum but not quite", then when things don't quite work out, you're on your own.
So, if you do exactly as Scrum prescribes and do not find success, in what way aren't you "on your own"?
Scrum "by the book" has a lot of material on the edge cases. How you handle almost every case that can happen in a software development team. So you can follow it like a workflow. Something unusual happens, you look in the book, do what it says. It's flaws are fairly well documented and understood, too. That's what I mean by being "on your own" if you don't follow it. At that point you can't just use a book to figure out next steps. You have to actually talk it out and use your brain to figure out what to do, because its unlikely your internal process has as much documentation around every single "paved paths".
It's also just a process. What does it mean not to find success here? If you ship broken software, it's unlikely to be because you didn't move the right ticket in Jira.
If you implemented Scrum "by the book", the likely "failure cases" are more things like people refusing to actually follow the process because they find it to be a waste of time, the overhead is too high, people are sick of looking at the book, etc.
Don't get me wrong: I very much dislike Scrum and you'll never see me push for it in a team. My point was merely that if you take an "On rails" experience like Scrum, and tweak a few little things, you lose on almost everything. The books no longer apply. When the entire point of this particular process is basically the books and documentations, doing "Scrum but not quite" really kills the point.
> If you implemented Scrum "by the book", the likely "failure cases" are more things like people refusing to actually follow the process because they find it to be a waste of time, the overhead is too high, people are sick of looking at the book, etc.
Yeah this is the nonsense propaganda I'm talking about. "If it didn't work, it's because you needed to do it more", which I think is absolute nonsense.
If you aren't following the Scrum practices, then you aren't doing Scrum. For better or worse.
It's like baking a cake without adding sugar; it's not really a cake anymore.
Sometimes teams don't understand practices and will discard them.
As an example - retrospectives. I've worked on teams where we inspected and adapted our process on demand. We didn't wait till the end of a sprint.
I've managed individuals on teams where they're discarded retrospectives. They've complained to me about certain processes (not scrum ones) and when I've asked, "why didn't you raise this in the retrospective?". "We stopped them, we didn't see any value".
No doubt they weren't getting value out of them, but that doesn't mean they were doing things well and they lost opportunity to improve as a group.
For context, currently, I manage a team of 60ish without Scrum. I see Scrum as a bit dated now.
> If you aren't following the Scrum practices, then you aren't doing Scrum. For better or worse.
Dogma.
> It's like baking a cake without adding sugar; it's not really a cake anymore.
Nonsense.
> No doubt they weren't getting value out of them
This piece goes against this piece:
> , but that doesn't mean they were doing things well and they lost opportunity to improve as a group.
Why would you continue to follow a process you don't find value in? If you didn't like a process, why would you feel like bringing that process up in a process-oriented equally-useless meeting? This is the dogmatism. "Well even though the meeting wasn't valuable you still should've tried". To what end?
> For context, currently, I manage a team of 60ish without Scrum. I see Scrum as a bit dated now.
Less interested in size, more interested in throughput/attrition.
You may think Scrum is dogma, but my statements are factually true by reasonable definitions of the terms.
>Why would you continue to follow a process you don't find value in? If you didn't like a process, why would you feel like bringing that process up in a process-oriented equally-useless meeting? This is the dogmatism. "Well even though the meeting wasn't valuable you still should've tried". To what end?
Strawman.
I'm afraid you'll have to take my word for it that the team where just bad at it. They got better with coaching and even managed to see some value.
The values behind the process are more important. This team just weren't talking to each other about their own performance.
> throughput/attrition
Of staff or customers? Attrition is very low of both. Not sure it's that relevant though. 3 engineers have left in the last 2 years and one of those three is returning. Is that a good thing?
As for throughput, even the slowest of teams ship on a daily basis. Not sure what other context you could mean.
> You may think Scrum is dogma, but my statements are factually true by reasonable definitions of the terms.
Your statements are dogma. "You must've just done it wrong" is scrum marketing. If it works it's scrum, if it didn't work it must not've been.
> Strawman.
Go on then, explain? Your gripe was that your team, that you admitted didn't find value in retrospectives, didn't bring up process issues in retrospectives. Why would they? They didn't find them valuable. Probably because the things brought up in retrospectives never got addressed. Agile solution? Add more process. Again, this is dogma.
> Of staff or customers? Attrition is very low of both. Not sure it's that relevant though.
If I employ 500 engineers and they're all miserable and constantly leaving, and we're never adding value to our clients, should I be proud that I employ 500 engineers? What does team size matter? Value-creation and employee happiness are the metrics that determine the health of an engineering org. Which I chose to label as "throughput", or how much value is created, and "attrition", a proxy for employee happiness.
OK, but scrum is supposed to be a way of doing agile. And "we're going to do agile by a rigidly defined process" is an oxymoron. If your process isn't one of the knobs you can turn, then you aren't actually agile.
>Most of Scrum's bad rep comes from people who think they know better, tweak it to suit their needs, then realize they made a ton of holes in the system and their Frankenscrum doesn't really work.
I remember when Scrum came out. It's a very specific method for a very specific company type that was sold to work with any company. Before scrum, each company kinda figured out their own processes based on their size, team composition, output requirements, and their risk tolerance. Ticket systems existed before scrum, feedback loops existed before scrum, bug tracking existed before scrum, prioritization existed before scrum. Scrum just took all the things that already existed and put them in a very specific and somewhat restrictive process. It was eye candy for the execs who had no idea how their department worked, it was a cash cow for the consultants selling it. It's ok if you don't have anything else or you're dealing with high turnover or something like that, but if you have a mature department already, it's probably more restrictive than not.
Anecdotally, I've experienced the exact opposite. The company I work with had been doing waterfall development for 15+ years. We switched to agile using the Scrum framework. It wasn't smooth sailing the entire way, there were teams early on that struggled with adopting Scrum. Most often it was due to the team attempting to solve every retro issue by updating "the process", which only addressed symptoms instead of fixing underlying communication problems within the team. The other major hurdle was getting product to accept that a roadmap isn't a set of commitments but is instead a list of prioritized "wants". Now, we're 3 years out from the switch and most of those pain points have been removed. It's lead to massive improvements in quality, predictability, and overall happiness across the company.
I had the same experience, in a longtime waterfall company, that adopted Scrum, with significant pain points but it was still a world of improvement. The thing that the Scrum haters don't see is that its biggest benefit is invisible -- it's in what doesn't happen, in what Scrum avoids.
Scrum avoids the eighteen-month waterfall death marches that kill a department or a company. Scrum gets you thinking in terms of managing scope, of pacing deliverables, of thinking of a list of prioritized "wants" as you say rather than a dictated set of commitments. Scrum makes you approach development as it has to actually happen in the real world, rather than as a dictated plan that inevitably won't survive contact with adversity.
Of course it's completely possible to still mess that up, with overzealous adherence to process and ceremony and metrics. It's very possible to run Scrum as a set of perpetual two-week waterfalls -- and very possible that that's still a great improvement over two-year waterfalls.
The problem with scrum discussion is it's always compared to waterfall, particularly unyielding waterfall. Even construction planning, which is where waterfall came from, of has slips and adjustments. There were other methodologies other than waterfall before agile or scrum arrived. I went to college in the mid 90s and they taught that waterfall existed but a more iterative approach was actually used in practice. If you were doing immutable waterfall, anything would have been better, maybe including nothing at all.
Scrum is like a global variable or a goto statement. If used carefully it could provide benefit but is often thought negatively. Used by untrained or junior team members and projects spiral. Senior members shy away because of experience.
People say that about communism too. I'm not open to debate any of the tankies I know are on HN, but if a paradigm brings utopia if implemented properly, and misery otherwise, and there have been no known utopia-yielding instances of the paradigm despite repeated attempts, it's time to consider that maybe misery is inherent to the paradigm.
You're not wrong. Communism implemented perfectly would also work beautifully. It just can't.
The main difference here is that Scrum is not nearly as complicated to implement. Though the issue is the same. There's always someone somewhere who thinks the process doesn't apply to them and the whole thing falls apart.
I did work in one company that had a fairly well implemented Scrum, and it did work pretty well. I've never seen it replicated though, so for all practical purpose, it's exactly as you say: it's not worth trying because it only works when done perfectly, and that almost never happen.
I thought one of the interesting points in the article was that Scrum gives a team some breathing room when there are many "stakeholders" in the organization wanting feedback, updates on progress, and to frequently change requirements.
The Sprint gives a defined time period during which the team is allowed to work uninterrupted by these kinds of requests, with the updates, feedback, change requests etc. taking place at the Sprint boundaries.
A "scrum course" is to Scrum what a code boot camp is to programming. You're going to learn basics but you won't have the depth of knowledge to handle complexity in any form, nor how to answer questions about Scrum that inevitably are asked by leadership. There's a reason the Scrum leader (or master) role exists. It's supposed to be the person who has a depth of knowledge that goes beyond a simple "scrum course" and can guide an organization when complexity arises. Organizations rationalize why they don't need to hire someone into this role because a Scrum leader isn't a producing role. And they're right that Scrum leader isn't a producing role, it's a role that's supposed to make producing roles more efficient, a force multiplying role.
Pretty sure there isn't, otherwise FAANG would enforce it top down. They already enforce other practices top down like hiring etc, so if they had data saying that mandating Scrum would make the company X% more efficient they would absolutely do it.
It might be a force multiplier in other types of companies, but it probably isn't in FAANG style companies. Similar to how FAANG hiring can make sense to them but still be bad for typical companies.
I mostly agree with this perspective. Alternative agile methodologies may fit better with certain teams (e.g. Kanban, lean, etc) and so having a top down prescriptive method of "agile" doesn't make sense. Another perspective is that Scrum is agile training wheels. Once a team can discipline themselves enough to follow a Scrum workflow successfully for a significant period of time, it shows they are ready to follow a more flexible agile approach and therefore may no longer need a Scrum leader guiding the team.
> They already enforce other practices top down like hiring etc, so if they had data saying that mandating Scrum would make the company X% more efficient they would absolutely do it.
lol, as if their top-down hiring practices are at all driven by concerns for company efficiency.
Why do you think otherwise? The whole reason they do it is that they looked at other big companies and noticed that everyone ended up with mediocre workforces after a while, so they did this to avoid that scenario. Then as they grew bigger they could hire a lot of data people to analyse different methods and see how well different things works to optimize further. When you spend tens of billions on engineering salaries at a data driven company you will do that.
I suspect the prescriptivism and detail have one end goal and that is for someone that has no idea how software is done to follow the procedures. It also turns the process into an "almost predictable process" for the higher-ups to see turned into a graph in some ppt. You also have to deal with people who needs to be told which shoe to put in first before they think the process is "confusing".
Same reason for PMP, it's to turn the procedure into "paint by numbers". Of course it rarely works that easily.
But of course you can't justify the multiple middle-managers and why can't the people at the bottom just "turn the wheel faster" without it so there's a bit of purpose in that.
> It also turns the process into an "almost predictable process" for the higher-ups to see turned into a graph in some ppt.
This is not just a scrum problem. Any agile process eventually reports to a layer of management that doesn't think in agile terms, and wants their PERT charts or whatever. The impedance mismatch at that boundary is one of the biggest difficulties with implementing agile.
The issue with all of this addition process, ceremony and meetings is work still needs to get done. Sadly the only time to do work is often outside of working hours since then, finally, the meetings and ceremony have ended.
I don't think this is the case. A two week sprint should have about 100 minutes of standup, 1 hour of planning, 1 hour of refinement, half an hour of review and 1 hour retro, ish (I can't remember the exact timings). About 5 hours every 2 weeks. Saying that adds up to a full 2 weeks of work is just... pointless. So falsifiable.
I can only speak from personal experience (perhaps our Scrum consultant did it wrong). We have a large group and even running through the all of the demos (mostly slides really) from all of the teams at the end of a three week period took days. Added to this the retrospective, multiple planning sessions as well as all of the meetings that we used to have prior to doing this.
In the end, it was just so obviously unworkable to everyone it had to be stopped.
However, you are absolutely right, we were not in meetings 100% of the time - it just felt like it. Many people need good swaths of uninterrupted time (perhaps 1.5-2 hours minimum) in order to be productive. Rapid context switching between various meetings and development work can be expensive. It is unfortunate when this time can only be found outside of working hours.
My suggestion is to start with no process and no regular meetings and carefully add process/meetings as required. If you need to talk with one or more people... call them right now (or arrange a meeting with a clear agenda - right now). All overhead should be carefully evaluated in terms of its actual ROI.
> We have a large group and even running through the all of the demos (mostly slides really) from all of the teams at the end of a three week period took days
I have several product teams (about 25-30 engineers depending on how you count them) on different products and we often squeeze a "show and tell"-style sprint review, where it's much more broadcast than interactive mode, in 30-60 minutes.
I think a general rule is if it feels like death then bring it up in the retro and change it. And if you can't change it, even with good reason, guess what - turns out someone's disallowing agile :)
I agree with all of that. To me Scrum is a reasonably minimal set of meetings that involve everyone who needs to be involved as little as possible to get the job done, but that won't be everyone's experience. Partly because Scrum is a methodology with a pretty specific use case; it's definitely not for everything.
I see enterprises commonly mess up the planning stage.
They take the waterfall approach, what do we need to get done by X to achieve goal Y. Then apply tasks/actions to get it done to people who are at a meeting. Following this step they shift over to scrum sprints, routines and measures to give management comfort on whether its track or not.
The alternate might be, what is the highest value items that we can achieve with this group of people.
Generally speaking, if the thing that is meant to help you get to value becomes more of a focus then you're in a spot of bother
I fully agree, but I feel you vastly underestimate the difficulty of having a team that more or less adheres to these few principles. In your average Fortune 500 corp, it's next to impossible to have it without prescribing. If scrum is just writing down common sense in a onepager, then that's actually useful stuff.
In my experience, junior developers like the rigid structure and senior developers would prefer more flexibility.
Companies that implement agile to the letter typically treat their developers like consultants. You're treated as a mindless drone that is supposed to work on tasks that were predefined for you. You might as well be outsourced.
I found that kanban-style works better imo. A prioritized backlog where you pick the most important task to get done makes for a more relaxed and friendly environment, compared to an environment with a rigid deadline every two weeks that everyone sprints towards. As long as there's progressed made towards the most important items, everybody is typically happy.
Every company I've worked at that followed some sort of rigid scrum process has suffered from burnout and general failure in one form or another
It treats developers like consultants because it's really designed for agencies working on one-off short-burst projects ( this is the only setting I've seen it have a positive effect )
by nature it just chews people up if it becomes a day-to-day practice, the whole process revolves around the assumption that there's lack of trust and team cohesion
A mandatory meeting in which all developers must be present and regurgitate status updates to the entire team, as an example, assumes this information wouldn't get to relevant parties organically and all team members must consume all status updates.
Which goes to some assertions that have been made about this thread: Scrum is a tool to build standardized minimums in support of mediocre talent. Elite tech companies don't use scrum because they don't have (as much) mediocre talent.
I don't work in civilian software development, but I do work in a setting with clearly defined, regular meetings/briefings. In our org, information frequently DOESN'T get to the relevant parties organically. There are representatives from every functional section; bringing every last team member in would be pointless and unsupportable due to space constraints anyway. The briefing is directed to the Commanding General (or whoever is receiving the brief in his stead, like a VP-equivalent in corporate terms). Our briefs are an opportunity to give the senior decision-maker 1) a snapshot of exactly what is going on right now 2) demonstrates to the decision-maker that the staff understands his/her mental image of the organization's direction/goals 3) forum for directed tasks/information requests from the senior leader to either Subject Matter Experts or the people who know them.
i've been a proponent of todo and task lists for years now. not one ounce of it feels like planning/prioritizing poker. i just throw a task on my list. i gauge it's priority by how pressing it is at that time. if it's super important, i'll probably make a calendar item for it as well.
do you know why i do this? cause i'm lazy. i can't be bothered to remember things or talk to people again so i do the one thing people often forget we can do: write it down. know how many times i've annoyed my lead by having to re-ask questions for a typical project? 0. cause i ask it once and write it down. it literally takes no time at all. it saves on having to do more meetings down the road because i have it all documented.
> For complex projects that span several teams across different offices and time zones, leading such a project is a full-time job for an engineer. Big Tech pulls in Technical Project Managers (TPMs) to manage these complex, and often strategic projects, taking the load off engineers.
> Dedicated Program Managers or Project Managers still exist within Big Tech. They are typically tied to external-facing commitments and customers – such as a Program Manager for the Apple partnership – or to long-running initiatives, like a compliance program. Similar to how TPMs are not allocated to a single engineering team, these Program Managers or Product Managers also don’t tend to have an engineering team, but they work across many teams instead.
> Engineers are encouraged to interact with the rest of the business and build relationships with non-engineers. In contrast, traditional companies often make it impossible for developers to interact with the rest of the business.
In my experience “traditional companies” will often have a bunch of people in cushy “gatekeeping“ jobs whose main function is basically forwarding emails back and forth between devs and the business. If you try to get direct access to the business usually the business is quite happy but the gatekeepers get very upset.
I work in a "traditional company" and the gatekeepers were installed to keep developers from going insane. When we have too much information back and forth between business and engineering, the loudest complaints from the business side (where the $$$ comes from today and tomorrow) drowns out the team's and department's internal goals, which are usually much longer term (meaning the $$$ is in months or years, not next week).
I believe this is why the author noticed that certain types of companies had engineers say the product owner was one of the reasons they're more satisfied. From when I started at my role, my team went about 2 years without one, and then within months of having one, we have been much more productive and have managed to create much more immediate and future value for the company.
The trick big tech uses to solve this is to not just pay extra for above average engineering talent, they pay extra for above average talent in all areas. So the people you talk to are almost always pretty reasonable and understanding, they want things done and complains but they don't ask for the impossible or unreasonable.
I think it’s deeper than that - these companies place product/engineering at the centre of the business, rather than at traditional companies that still see it as a cost centre.
Product managers and UX engineers are there to do product management or UX design, not to filter information from the rest of the company to programmers. That means that most of sales requests are better forwarded to a product manager or UX designer, but there is nothing stopping them from directly filing a bug report to a developer and chatting about potential fixes/timelines etc. Also if some feature is important the product manager can just tell them to talk directly with the relevant engineer to get the feature done.
This is because businesses chronically underinvest in developer/engineering resources.
Developers tend to be viewed as something that is not "core" to the business, and generally only necessary on a project-by-project basis. So, projects are initiated, resource needs identified, and, since nobody wants to hire a bunch of full-time staff, most likely a consulting firm or off-the-shelf product is selected and configured.
The downside is now you have a thing that needs to be maintained, and probably it will be maintained by a limited pool of "in house experts", who are also responsible for just about everything else.
The net result is that you've got this centralized resource that is under provisioned and, well dear reader, what does your training as a systems engineer tells you will happen with a centralized resource that is under provisioned? Contention? Rising latencies and queue lengths? Disastrous to recover from failures?
You think they would be taken aback by the cost of a temp developer. In my case we easily charge the salary of two or three junior devs for one person and we are very sticky...not one contractor was out of work during the pandemic while those companies laid off FTE in significant numbers.
The problem arises when the gatekeeper either no longer understands or no longer cares why their position exists. Then they just become an obstacle to the very communication they are supposed to be there to facilitate.
I've also seen some instance where gatekeeper were pretty effective at filtering users demands because some of those requests were too dumb and the "paying" users were used to have everything they wanted, and because they "paid" the dev department, "they had to do everything they wanted".
Like asking for a 6months dev work to help them save 1 hour annually on an annoying task they had to do.
My experience has been the exact opposite, if the developers understand what the business is trying to accomplish and what the incentives are then they are in a much better position to prioritize and build the right thing. Every single time I’ve seen 6 months of work on a useless or low value feature it has been the result of the dev team getting requests via some PM telephone game.
This is a tricky balance. Developers can also get overwhelmed with requests for estimates, automation tasks, etc etc, and get annoyed that people have so much contact with them. But it is a tricky balance; there's no obvious solution.
Sometimes it's helpful to have some division of labor between field/sales/support engineers who go to handhold customers, understand their particular problems, and prototype fixes, and "product engineers" who have less customer interaction. This is somewhat similar to the SRE/SWE split at places like Google.
If I were designing an organization with such a division of labor, those wouldn't be different job descriptions but different activities within a single job description; maybe I'd spend 22 days a month doing product-engineer stuff like adding features, and 3 days a month doing support-engineer stuff like visiting customer sites, diagnosing customer problems, and answering calls, while maybe hatchnyc would prefer to spend 22 days a month doing support-engineer stuff and 3 days a month doing product-engineer stuff. The reason is that, doing either activity, you gain an enormous amount of knowledge that's crucial to the other activity, but very difficult to even put into words, much less into a knowledge base.
> Like asking for a 6months dev work to help them save 1 hour annually on an annoying task they had to do.
I see this as a complaint a lot, but... in a private company, they're the ones writing the checks. So if they want to spend hundreds of thousands of dollars to save an hour, that's their prerogative. It's certainly our obligation to point out the cost (including ongoing maintenance) but again in a private company you don't really have the option of saying "no" except with your feet.
Yea, I don't see the complaint here. It's part of my job to let whomever the client may be that what they're doing may be a bad use of resources (typically in a documented email in a very positive tone with lots of "decision makers" on the email, also with a nice lead in explanation as to a way it could be efficient under some set of circumstances so they can copy paste that as their "well we weren't sure" pathway forward).
After that, I really really don't care anymore, that's someone else's problem. I've played that soapbox and it's a waste of my time and energy. I make sure responsibility is passed back, documented and proceed. If you want to throw hundreds of thousands or millions at me and whomever to some wasteful request, go for it. You have the option, you've been warned, and I've given you a valid excuse to proceed with high risk, high uncertainty of ROI option with everyone important involved.
I do a lot of contractual/consulting work so even warning can mean early termination of work forward, so I'm taking a risk even informing you (some people actually listen and work stops early during consulting time and development follow up never occurs, these are the smart people and they're bad for my livelihood)--I could just do whatever you ask and get paid. If I were salaried, there's even less risk of me losing income forward by informing so I say in those cases just do your professional due diligence go let people know it's a bad idea without creating political land mines for anyone, then allow them to step on land mines at their own discretion.
There are other factors at work that may exist outside of the "1h of labor saved" too.
It could be something that's tricky. While the optimal case is 1h of work, if something goes wrong then its doing forensic auditing on the books in 3 months and spending lots of time there.
It could be something that needs to be repeatable. Yes, it's 1h, but before this was done Alice did it and before that Bob did it. They did it slightly differently and that cause problems. Having a computer program do it provides change management to the process of doing that task.
It could be something that needs auditing. Tying into the previous two, having a "this is how it's done and all the calculations that go into it along with a test suite" makes it easier to be assured that it is working correctly.
There are lots of things outside of the purview of a developer working on doing whatever task needs to be done to solve a problem. The value of the problem being solved is the issue of the manager or the customer.
> in a private company, they're the ones writing the checks
As long as they're being otherwise reasonable, I don't have a problem doing something that doesn't seem terribly "important" to me either. However, in my career I've had many frustrating instances where they were demanding something in a timeframe that I couldn't deliver it without ensuring that there wouldn't be any unintended side effects - and then blaming me for the side effects when they came up. I only push back when they don't realize how complex what they're asking for is. Of course, then they play the "tell me how long it's going to take so I can argue with you until you agree that it will take as long as I originally asked for" game.
> they were demanding something in a timeframe that I couldn't deliver it without ensuring that there wouldn't be any unintended side effects - and then blaming me for the side effects when they came up.
Far too few developers understand: just because the people writing the checks ask you to do it and you implemented it exact as they ordered you to doesn't mean the people writing the checks will take responsibility for the resulting bad situations that can lead to them not writing you checks anymore. If anything, you'll be used as a scapegoat to prevent them from not getting any more checks written.
Of course you have the option of saying "no" unless the company culture / leadership climate does not allow it. That's far from the inherent obligation you're describing.
I do however agree with leaving any organization that does not allow you to express disagreement.
I think that depends on whether "no" means a nice way of saying "this is stupid" - which I agree with, and have done before, to mixed results - vs. outright refusing to do work that's been requested. I don't think an employee who is being paid a wage by an employer has the right to refuse to do something just because it's a stupid idea.
I'd much rather have an employee who will refuse to do work that is stupid than one who will do whatever is asked of them. Especially if it's something as bad as the GP example of six months of dev work to save one person-hour per year.
This is really only true of it's coming from the very top of the food chain. Sales is obviously an important part of the revenue stream, but it's not the only part and it doesn't automatically trump all other concerns. For example, it's not useful to sell a product that doesn't work, will cause huge customer churn and will destroy the company's reputation. If you're a shareholder or care about the longevity of your job, it's important to push back on wasteful behavior.
Absolutely agreed, I implied with "writing the checks" it was the actual owner(s) of the business making the decision but could have been more explicit about it.
I have seen it more often that gatekeepers filtered user demands based on their own knowledge and not on technical feasibility or effort. So you are told that a user needs a certain thing in a certain way but when you talk directly to the user you learn that the user actually had a different need which you can fulfill in an easier way than the gatekeeper thought possible.
In general I think there is a huge advantage if developers know first hand how their product is being used and not through gatekeepers.
Honestly that's the whole problem with Democracies. And we solve it with sampling and surveys. Probably if we can come up with a better answer, we will improve Scrum, and Democracies :-)
Unless you work on an internal tool "business ops" are not your users. I'm talking about real products with external users that are only really identifiable as an "account" unless it's a massive contract with dedicated sales team.
Even when the possible goals are identified, someone has to prioritize them at some point, since there are always more goals you could achieve than goals you will achieve. To prioritize, you need to know both the benefit of a goal (which is really a probability distribution, not a number) and its cost (which is notoriously a wide probability distribution before you have done it). Sometimes this involves trading off the needs of different "goal donors"; in XP these all filter through a single "goal donor", confusingly called the "customer", who decides how to order the cards in the box once they're estimated. The goal donor (who may or may not be the gold owner, another confusing name) can reorder the cards at the beginning of each iteration, either because new information has appeared about the benefits, about the costs, or about the possible goals.
It sounds like what you're describing is the lack of such prioritization. It isn't necessary to keep the developers ignorant of new user "demands" in order to prioritize them; they just need to be empowered to follow the priorities the team has chosen instead of accommodating every demand.
Sometimes it's easier to accommodate a demand than to write up a card and estimate it, though. Sometimes a bug fix or layout tweak is obvious enough that doing it takes only a minute or two; then, it's just a question of process overhead whether the actual cost of doing it is five minutes or five hours: potentially multiple forwarded email round trips, a fresh checkout, pair programming or other code review, compilation time, test suites, commit comments, merge requests.
Usually, in my experience, if you have extreme overhead, most of those changes never get requested, because they don't get prioritized the way they would with the order-of-magnitude lower cost a lightweight process can provide. The result is that they never get done, so the product is full of easily observable minor problems, which is experienced as shoddiness.
What a utopia this movie portrays! Individual cubicles, administrative assistants for senior staff, and the protagonist has only been asked to work over a single weekend.
Yeah, I learned the hard way that one of the worst things you can say at a megacorporation is "this manager is just creating work to justify their existence in the company". In my case, it was directly applied to a specific manager (who heard me say it), and it led to me being lectured and yelled at and told that it was "unprofessional" talk (which to be fair it kind of was).
I'm definitely not a protected class (I'm a tall white dude), and I am not being 100% fair; they weren't screaming at me or anything, just an elevated voice and very angry words. I don't believe they said any curse words, and even if they did I was somewhat sympathetic to why they would be upset with me (even though I do stand by the content of what I said).
It was probably still unprofessional, but probably not as bad as I made it sound...my bad!
At the last large traditional place I was at it seemed a lot like the upper management types had put these gatekeepers in place so they had people that would pander to them. The amount of times one of them told me they couldn't tell their boss some bad news for risk of getting fired or chewed out, but wouldn't let me go and tell them the truth was mad.
From what I could tell no-one was firing engineers because it was too expensive and we didn't really care since we could get more work within a few hours, and the "bosses" didn't like the lack of power they had in that dynamic.
The best arrangement is to have business report to business and engineers report to IT, but have them communicate directly. This was they can't be pressured by business into short term tasks, because they are on the same level and not just mere subordinates of business folks.
In traditional (not engineer-first) companies, engineers are subordinates of PMs or business directly without their own agenda, so they just accept the fate and become a feature factory for the company.
> In my experience “traditional companies” will often have a bunch of people in cushy “gatekeeping“ jobs whose main function is basically forwarding emails back and forth between devs and the business. If you try to get direct access to the business usually the business is quite happy but the gatekeepers get very upset.
“But I’ve got people skills, dammit!”—Tom Smykowski
Constructive / progressive organizations can be built or transformed into, where collaborative knowledge-work happens at all levels.
Scrum is a parody and pretty much anti-agile.
Traditional organizations are driven by the cost-accounting mindset that creates gate-keepers and makes it impossible to share a common purpose, let alone collaborate across teams, units or vertically.
My current company is like this. I work directly with the business people anyway. When anyone says anything I just point them to my output, quality, and the feedback from the business people - shuts them up every time.
I’m one of these hypothetical gatekeepers and would be thrilled if a dev could successfully take this off my plate. Just prioritizing the incoming requests from stakeholders is a full-time job.
I really want to say this: SAFe is an awful process and a trend that will hopefully go the way of Unified Process/RUP. I won't go into it, but it's largely created and popularized by a vendor to sell their software. It is poison and exists to keep its practitioners employed. It attracts the highest-ego PMs like moths.
I find it interesting that they author references Skype circa 2012. It sounds like classic "uppercase A" agile: they reframe success as shipping and hitting other invented agile milestones, while masking shipping lousy and incomplete product that is not succeeding in the market. That was right around the time Skype started being bad. It may succeed on some metrics because MSFT started bundling it, but anyone that actively used it at that time should know what I mean.
The idea of an "agile enterprise" is intrinsically absurd. Teams are agile, not companies, and teams and products should be loosely coupled. That's why the big tech companies in this link have converged to similar answers to this problem (they are not "agile enterprises" but give teams some latitude to solve their own problems, and rather focus on results). Kanban is better in most ways for most teams.
Also, Tuckman is a silly model that is only popular because it rhymes.
PS: Strawberry-Jam-O-Meter and a Wrong-Order-O-Meter and their descriptions could hardly be more cringe inducing. Skype should be a part of the curriculum looking at less than ideal acquisitions and post-acquisition execution. https://www.wired.co.uk/article/skype-coronavirus-pandemic has is a reasonable overview if you're not familiar. They lost the consumer market and failed at enterprise (see Teams) and Teams in turn essentially failed in the general market- where most places that were free to do so went with Slack.
As a Dev, I loved SAFE. It changed everything about how our program operated to the point our delivery became extremely predictable and we still had every 9th and 10th week to tinker on new ideas or do refactoring and cleanup where we wanted. That program purred like nothing else I've ever been a part of. Senior leaders sat with devs, started talking to everyone about their priorities - people we had never seen before we started doing PI events. Antagonism dropped. We went from not being trusted to deploy during an annual heavy use period to being totally trusted to deploy and given additional contracts for maintenance. And this was with the exact same people who had been there. We just weren't working very well before we got into SAFe.
I think one of the keys to this was the buyin everyone had. It wasn't totally consistent at the beginning, but over time everyone got on board, we did trainings, actually incorporated feedback consistently.
I hear SAFe hate all the time, and when I ask how things went, inevitably it turns out they never actually did SAFe. Somebody made them email a 10 week plan and used the acronym and that was it. If you're gonna do SAFe, do SAFe.
“Senior leaders sat with devs, started talking to everyone about their priorities - people we had never seen before we started doing PI events”
Pretty much every process will work successfully if everybody participates in good faith. I don’t think SAFe has special properties that people honestly work together. For example in my company it would just create a new bureaucratic nightmare where we would hire even more managers, project managers and consultants while senior leadership still would never talk to the lower ranks.
SAFe has roles. If you don't have people doing those roles, you're not doing SAFe. If you have people doing roles that aren't in SAFe, you're not doing SAFe. It's really quite simple and I don't understand the pushback. Do it or don't, but trash it if you didn't actually do it.
A general goes to the commander of a group of soldiers and says "In 3 months, I want them to be able to demo the "march in formation" feature in front of the big-wigs who decide whether to give us more funding."
So the commander says "Yes General, we'll do Drills™ to make it happen."
And then the commander writes down "Do drills" on a TODO list, gives presentations to the soldiers about the importance of drills, and maybe even hires a certified Drill Sergeant.
But the commander and soldiers never actually do drills. They clean their rifles, practice shooting, and do other soldiery things, but no drilling.
3 months later, surprise surprise, the soldiers walk all over themselves at the big parade, the general is angry, and the commander is confused why simply talking about drilling didn't work.
But lots of developers actually hates Scrum even when done correctly. The problem is that Scrum is very rigid, it tells you how long your development cycle should be, what your meetings should look like etc. They'd prefer to just do what needs done, have the meetings that needs to be had, deliver features when ready rather than have arbitrary deadlines (sprints) etc. I understand that many developers loves having that rigid process since it is easy to just go and do your work without thinking about the bigger picture, but lots of people wants to do the other parts and feel constrained by Scrum.
And no, "adapting the process for your needs" doesn't work. The problem is having the process mandating meetings and timelines in the first place. If you just do everything freeform as these people wants then it isn't Scrum.
That's a far stricter presentation of Scrum than it deserves. The scrum guide doesn't prescribe a sprint length any more specifically than "less than a month"; it doesn't say "you can only deploy once a sprint".
Now, it might be that some people selling Scrum have Very Specific Views about some of these things, but that's not Scrum either.
> it tells you how long your development cycle should be
No it doesn't. Refuting claims about Scrum that are definitionally wrong stops us actually having useful conversations about whether actual Scrum is good or not.
By your analogy, the purpose of doing Scrum (drills) is to get better at doing Scrum (parading) rather than doing actual work (soldiery things) - which I think pretty accurately sums up my experience with it.
As I understand the analogy, the purpose of scrum (and particularly reporting), similar to the purpose of parading, is to demonstrate that you have in front of you a large group of individuals trained to work in an organized, consistent and predictable way. The implication is that if given any other "somewhat similar" task, this group would be able to perform that task in a similarly organized manner too. The choice of what the task should actually be is then left to someone on a higher pay grade.
But this isn't actually what happens when agile and similar have gone wrong in my direct experience and the parent poster would note that this is the same kind of general "no true Scotsman" response to "it didn't work" that you get from self-help gurus. The system always works, all failures are that you did it wrong.
Probably because a lot of people aren't really doing it, for reasons that are entirely predictable if you've ever worked at a big company. Consider the core of the Agile Manifesto http://agilemanifesto.org/:
> Individuals and interactions over processes and tools
> Working software over comprehensive documentation
> Customer collaboration over contract negotiation
> Responding to change over following a plan
How many big companies do you know that are organizationally capable of valuing individuals over processes and tools, or responding to change over following a plan? Almost all big companies have the opposite value system on those two points, imposed from the top down by the CEO (with the exception that they do value individuals if those individuals are senior management). Many big companies also have incentive structures that strongly countermand the other points, too. Comprehensive documentation is crucial ammunition for redirecting the blame for bad decisions, for example. (Cf. https://news.ycombinator.com/item?id=28674388: "we not only need very thorough clarity on what feature development will yield the greatest returns, but also deniability that we had good reason for doing what we were doing if it turns south.")
A perfect example of using agile rhetoric without agile practices comes from today's post about what tech managers do—"standup" meetings where people don't stand up! https://news.ycombinator.com/item?id=28677250 This results in the meetings lasting half an hour, doubling or tripling their cost per participant, even before you grapple with the increased number of participants that likely results!
I have my doubts about how effective Scrum could be even if practiced perfectly, but Jeff Sutherland, Mike Beedle, and Ken Schwaber are among the authors of the original Agile Manifesto (not just the signatories!), so at least they endorse those four core values and tried to make Scrum reflect them. So I think it's justifiable to say that if you value processes and tools over individuals and interactions, or following a plan over responding to change, you're not really doing Scrum, or any other agile process.
This. You can recognize the agile cult members by how they respond to failed projects. No matter why they project failed they always say "we need to be more agile next time".
> Why is it that every criticism of agile/scrum/safe is met with "well you just weren't really doing it?"
Not every criticism. Just every criticism that is actually people just not doing it. "Standups become status update meetings, therefore Scrum is stupid" is an example.
In the first PIs after we started moving, this was the case for us too. But it got better every PI. The last few PIs I was in had 10 teams and a couple of them would be playing catchup, but most would be working on other unplanned projects. Either refactoring or learning or getting extra things done.
As with all "Agile Frameworks", they are marketed as "If it worked it's because you did it right, and if it didn't work it's because you didn't do it right".
Is something actually valuable if it is A.) Inflexible and B.) Rarely goes well?
These things are tools to be used. If you use focus on the tool and not the end user, things will go badly. Typically you will see teams that are very hyper focused on some particular metric (time, points, stories churned, etc). Yet not focusing on 'how do I get my customer what they need'. These tools help devs break down the tasks and tell the end users 'hey we only have this amount of time so focus!' If they become about metrics or just randomly skip steps that depend on each other, you will fail.
You'd have to trust me, but I know SAFe well in theory and practice. I still don't like it, but I am glad you've had a good experience with it. It may be that it's the right fit for some companies. It is definitely better than organizational dysfunction. It sounds like your environment was dysfunctional and bringing SAFe in fixed that, at least.
> The idea an "agile enterprise" is intrinsically absurd. Teams are agile not companies
I’m really not sure what you are basing this on. Agile, as I’m familiar with it is framed in terms of lean management which is a well established approach to running business, exemplified by Toyota. Now you can debate the suitabilities of this and the effectiveness til the cows come home, just as you can with agile and scrum etc but it isn’t accurate to say that there aren’t a lot of businesses that aspire to the lean approach.
I don't think it's fair to say that http://agilemanifesto.org/ is "framed in terms of lean management", or indeed management at all. While there's nothing making it impossible to organize a company along those lines, you may be interested in https://news.ycombinator.com/item?id=28677886 where I explain why the Agile Manifesto values are deeply opposed to many established company value systems.
My sole experience in Big Tech consists of Facebook and the post more or less matches what I saw there. It seems to me, though, that the author views "no process" from a very positive lens, with no discussion of negatives. Like how at FB so many teams use spreadsheets to track their work (sometimes multiple spreadsheets per team, sometimes no tracking at all). There is some internal tooling, which is quite basic and at the time I was there it got deprecated without the replacement being finished yet. Whatever you say about JIRA, it can handle complex projects way better than a spreadsheet.
My biggest issue with the whole thing, however, boiled down to the set of incentives that centered around individual performance. You see, the teams weren't _teams_ per se, but a collection of individuals working on their own projects, sometimes related to those of other teammates, sometimes not. A team-oriented process like Scrum or Kanban cannot survive in an environment where every person is optimizing every decision for their advancement/bonus/whatever. There's some exaggeration here, but I definitely saw a lot of this at FB. Having come from a company with high-functioning Scrum and Kanban teams that worked together to achieve a common goal, I'd choose that any day, JIRA or not.
This is obviously a single example and many of the issues were Facebook-specific. But I bet there are other, not so positive, sides to the "no process" story at the rest of the companies.
Note I'm not claiming that the Agile processes are by definition good. I've seen plenty of bad implementations as well. At the end of the day, every group of humans is different and may require a different process (or no process at all) to maximize their success. What I'm suspicious of is the "every team picks their own process" claim, having seen company-set norms exerting enough force to make deviations from the common pattern rather painful and counter-cultural.
Funny that you mention spreadsheets. When I worked at Microsoft my entire business unit (multiple thousands of engineers, managers, PMs) exclusively used Excel for project management. Clean UI, filters, data rules, conditional formatting, validations, pivot tables..it was the absolute best. And with Excel data tools it was all hooked up to a central database (TFS) for live sync/updates.
Biggest difference between big tech companies and smaller companies is the sense of urgency to ship something to market and the time scale for that urgency. Bigger companies have lesser pressure to ship things too early and have slower (larger) time scales for product release cycles.
Both in enterprise space or in consumer space, smaller companies are on a much more rushed timeline, for varied reasons including but not limited to financial situation of the company.
Bigger companies can afford to build more slowly. This affects how the company plans and executes as well as rewards performance.
In a smaller company, when a complex thing has to be shipped on a shorter time scale, a lot of divide and conquer and quick coordination across large number of contributors is required. This is a necessity.
In a larger company, deep complex things are built by very small teams or just individuals over a relatively prolonged time. Individual engineers prefer to keep chiseling away at a particular problem until they can showcase a significant impact with high difficulty/complexity of the problem/solution. That's the consequence of individualistic performance measurement culture.
Both cultures have pros/cons and there are rotten extremes in both.
I had the same experience at AWS. A team of siloed engineers working on projects for the same product, with very little collaboration. Projects were tracked in spreadsheets. Time estimates were created out of thin air to meet the desired (impossible) deadline.
This kind of mass delusion has been pretty much de rigueur my entire career (decades):
Software is typically quite hard and mostly unknown at the beginning of a project. Most of the hard work is figuring out the things that weren't known. It's a learning/discovery experience. It's essentially impossible to predict how long that activity will take. The further out in time you consider, the less likely you'll have much of any clue about the subtasks to be done. And of course in the meantime the participants likely aren't even working close to 50% of their time on the new project -- instead they're fixing the bugs and technical debt in the last project.
And yet, the "stakeholders" simply wouldn't accept any narrative like the real one (we kind of have an idea how to do this but we're going to figure it out as we go and it'll be difficult and unpredictable). They'd go hire another bunch of folks who are willing to salute the flag. So instead we pretend that what we're doing is predicable and plannable, like building a bridge that's a bit longer or shorter than the last 10 bridges that we built.
The stakeholders probably have a pretty good idea that the developers are making it up as they go. But everyone behaves as if the reactor isn't on fire.
Have you spent the decades of your career working at companies that blew their competition out of the water with better software, like Google, Apple, WhatsApp, and Facebook, or at companies whose software was unremarkable or inferior? If it's been some of both, did you note a difference in the degree to which stakeholders grappled with the kind of uncertainty you're talking about?
Very interesting, do you have any examples of how FB manages long term (2+ years out) project that involves 100+ Engineers? Are those 100+ engineers all "individuals working on their own project"?
I worked at FB for a while, and the problem with long term projects were that they were long term. In FB you need twice a year to make self review that shows what kind of impact you created during last 6 months (and based on this you are rated, raises, refreshers, etc). In long term projects impact might be only at the end of project, which leaves you "impactless" for a long-long time. Because of this people didn't really want to join any long term projects.
While I was there management was trying to resolve this situation and transmit message that long term projects are impactfull and important (and strategic to company), yet they couldn't answer question of how impact should be calculated every 6 months.
Maybe by now they came up with some formula or something
+1 to everything simplyaccont said. Long term projects were a tough beast to manage given the short-term incentives. Usually those got done by breaking them up into milestones and hitting those. That said, my general impression (as a first-level manager who sat in on calibrations) was that unless you move some serious metrics in those milestones, it would be difficult to exceed expectations and much of the upper management encouragement for long-term projects was mostly empty air due to the 6-month performance cycle.
That said, large cross-org projects certainly do exist. To answer your specific question, there would normally be some sort of a lead on the whole project, tech lead or PM lead or, often, a pair of those. They would meet regularly with leads of various sub-areas, organized similarly, and ensure things line up properly. Now repeat recursively until you get down to team level, where there might be a tech lead, or perhaps just a senior engineer working with a few non-seniors on their portion of the larger initiative. At this point this would get broken down into individual projects that each person owns and is accountable for. Tech lead may or may not contribute code to this project, they (and their team at large) might be involved in a bunch of parallel initiatives that often don't have much to do with each other, other than the product scope the team owns. Things like daily stand-ups don't work when everybody has their own work stream (and often multiple streams at that).
Hope this clarifies. The setup works for certain types of personalities, but I found it to be a very individualistic culture that I didn't particularly enjoy.
At Shopify, we tend to do whatever our engineers want in this regard. My team meets weekly and looks at a kanban project board. If we need to adjust, we have retros, etc and change the process. We have the autonomy.
In a past life you would be told Agile meant “self organizing teams”. But in practice that was only allowed in a narrow definition of change under the prescribed process being foisted on teams from above.
IMO while you need some consistency to get alignment on goals at a high level and coarse quarter-level goals, at the team level you can more or less let the team decide and then judge them on their effectiveness.
Odd how so many companies want to do “what [successful tech co] does”. Yet those companies innovate their own processes.
Nice to see big companies working like this. I wish more places would understand that different approaches work for different people, teams, products, technologies and contexts. There is no one true way, and most attempts to impose one tend to create something that is bigger and more complex than any single team needs. It's like the human equivalent of a code library that is trying to solve too many problems and so becomes an unweildy mess of config and options and meta-problems.
We (Bugsnag) are a relatively small engineering team, so have the advantage of low comms overhead, and we do have an overarching approach, but each of the teams works a bit differently. Even from project to project I'll adjust what makes sense based on complexity/risk/size.
I find this concept utterly baffling, but probably because I've never worked at this kind of org. I can see how letting engineers run their own process is great for engineering efficiency, but how do they know what to deliver and when?
My conjecture (please confirm or deny) is that these self-organizing teams are a result of and not a cause of big successful companies. You are probably iterating on a very well-understood and successful business model with a huge cushion for mistakes because you have some much revenue.
By contrast, I have mostly worked in consulting and non-tech orgs. If we don't show that we're spending their budget on high value features, we get layoffs. Hence we not only need very thorough clarity on what feature development will yield the greatest returns, but also deniability that we had good reason for doing what we were doing if it turns south.
I can't speak for Shopify, but in general you want teams that don't need anyone to tell them what to deliver and when because they can be trusted to figure that out for themselves and make a good call. That can't be done without having customer representation on the team (which you should anyway), and having engineers capable of taking a customer viewpoint. That is what "self-organising" should mean, but few organisations are mature enough to transition to it.
If you're in a situation where you need to show you're "spending their budget on high value features" that's a low-trust feature factory being treated as a cost centre by a remote client, not a value-producing unit setting its own terms.
There's a vast difference between being a cost center and being willing to set money on fire. Our teams have a lot of autonomy or at least influence in product strategy but it's because we empower our product managers, designers and researchers to help business understand how to achieve their goals. A lot of these projects are greenfield where the team comes in pretty blank on what needs doing. Developers are part of the process too but they don't generate much insight into what customers want. Only how to execute.
> Developers are part of the process too but they don't generate much insight into what customers want. Only how to execute.
How could you fix that? Would it be sufficient to give the developers more contact with the customers, or would you need to hire different developers, for example domain experts who also knew how to program?
I ask because, when you're writing software, you're constantly making tradeoffs between different aspects of quality: throughput vs. latency, flexibility vs. performance, learnability vs. routine usability, information density vs. skimmability, recoverability from errors vs. security, predictability vs. everything else. The more you know what your customers want, the better you can make those tradeoffs.
Ultimately "how to execute" is the automatable part of the job.
There's nothing to fix it works great. And it's worked this way at every company I've ever worked at. As hard as it is to find devs who can do frontend and backend you can't find any who are good coders and also know how to generate customer insights and spend hours doing interviews and requirements gathering. They're different jobs. When it comes to actually designing a solution then tradeoffs on implementation approaches are definitely done with the dev team.
What youre stumbling on is these “big tech” cos are growing horizontally when possible, and its normally local engineering roles that used to seed build and manage new teams. Thats why there some pretty extreme reactions to promotion process or senior role interviews. Yes, knowing how to organize a team, mentor junior developers, do market research, evaluate customer feedback, model marginal cost, and write a business proposal are key skills over the lifetime of those senior ICs. Yes, I could backstop those areas for my team mates today. But once we succeed each of those senior developers on my team will fork off to their own team and repeat the process, thats how we scale out.
A corollary is these self organizing teams add up to self organizing business units. The teams know what to build because theyve at read & internalized what matters to the business. And senior members should be involved with the 6-12 month planning cycles that happen across that business unit. So their day to day execution is informed by, and happens in the context of, the larger business. Think of something closer to hierarchal federation than directed work silos.
WRT to “lighting money on fire” there are teams that look like that. But thats generally a speculative investment with a medium term (~2-3yr) goal theyre working towards. This is business, so its not free or infinite, and in a sense their efforts are competing with the alternative efforts that could be more profitable.
Then that's the difference. Siloing devs away from the customers is something you see as a positive. Which, for some situations, it can be. But if you're in a situation where there's meaningful advantage in cutting the feedback loop down and reducing the number of handoffs between functions, it's not.
The key here is that you're using the third person: "their goals", "their money". "Business" is a remote third party, distinct from your teams. It might work, and a lot of people do it that way, it might be the only way your business model can work, but it's a long way from optimal.
We're absolutely not siloed. It's a question of expertise and attention. And also we're media so we have millions of users with extremely varied habits and needs. Discovering what those are is a huge job and one best done by a dedicated expert. I can't spend my dev hours having them sit and listen to stories about business process and sales funnels when there's dev tasks to do. Unless your company only has a single job title where everyone does everything, then I don't think we're doing anything crazy.
The answer to your original question is "trust the team". Integral to that is that they must have the skills on the team (or close to hand) to be able to do specialised work, in the same way as you might have someone with the "DBA" job title who specialises in databases. What you would expect is not that all the database work goes to the DBA, but that other team members would pick up DB jobs that don't necessarily need the full depth of their skills, with the DBA there to help if needed, and the team would be able to sort all that out themselves without management overhead. Incidentally that's also why you don't need to hire people who already have cross-cutting skill-sets. You're aiming to hire people who can learn, and trusting them to do so.
If you're not in a situation where you can envisage "trust the team" being something you could live with, then yes, what you're doing probably looks rational to you. "I can't spend my dev hours..." makes me think you're in a fundamentally McKinsey/Taylorist environment, low-trust by design. That's not uncommon, but it doesn't make other approaches wrong either.
One thing I learned a long time ago is that a "bad" process that is followed universally by everyone (dev, pjm, pm) will always out perform a "great" process with only token buy-in and constant exceptions.
I was there when Agile was invented. In my opinion, Scrum has always been the worst embodiment of a good idea.
In terms of content, teams of experienced developers have always worked in the spirit of Agile (of course there are exceptions). This informal understanding was and is superior to a formal horizontal Scrum implementation. For example, it preserves seniority and true accountability - two things that Scrum pretty systematically destroys in my experience.
Initially, I was hoping for additional solutions to problems in the vertical direction - management, customer relations, etc.... But that never materialized, at least in my environment. And since I've been in the industry for more than 20 years, that's not too little.
These days, mandatory Scrum is a contra-indicator for any project that crosses my path as a freelancer.
It seems to me that the industry's original sin with Agile was treating it as anything other than a model for managing a team of software consultants. There are certain types of internal product teams that can meaningfully emulate this paradigm, particularly in a B2B context where a tight feedback loop can be cultivated with customers. Chances are, though, that product teams, particularly in DTC businesses, don't have close enough relationships with their end users to be able to practice Agile as written.
I always viewed Scrum as training wheels. I've imposed it on teams who were dysfunctional as a way to get them on the rails. Hands and feet inside the car. Child safety locks engaged. It's a bit patronizing, but if you've ever worked with an unengaged team, it can be necessary. If you do it well, the team gets the feel for what agile delivery feels like and don't need the rituals to know what kind of touchpoints are actually required. Like the bottom of the Dreyfus model of skill acquisition. If you're "Big Tech" and have only hired the best and brightest and are working on enthralling problems, you tend to get people at the top of the pyramid who just run agile even when you don't tell them to.
I disagree with 2. Now, perhaps you disagree with 5. These were written in 2001 for problems in 2001. Now, some of these problems have been solved. Some of them are taken for granted today. However, we take them for granted in large part because a group of people got together and said, "we can do better."
I personally think it's time for another group (not the old agile luminaries but people with new ideas) to take a look at today's problems and take a fresh stab at it. Maybe it looks like "Plan-Build-Ship." But maybe it looks different.
But first, I think we need a clarifying question: What in the principles or manifesto do you disagree with?
I disagree with the whole thing. It was written by consultants to sell consulting services. The "principles" are empty aphorisms that have practically crippled the software development industry.
Any method/framework that is based on a "Manifesto" is despicable. The Manifesto gives the sense of entitlement of a moral standing. The principles become moral principles, and produce a Manichean divide between the new (good ones) and the old (bad ones), undermining any hope of truly collaborative work places based on trust and purpose.
Rather than moral principles, I prefer to base all discourse on scientific principles.
And those moral principles, deemed to be good, can be taken as assumptions. Assumptions can be questioned. Improved. Even replaced.
A manifesto and the holy writings like the Scrum Guide are unquestionable and give rise to dogma, tribalism, and all sorts of psychotic reality distortion perspectives.
If you're tired of Scrum, Manifestos and Holy Guides, there are better alternatives which foster much more humane, sustainable and collaborative working environments. I invite you to check out my TameFlow Approach.
Reading this, I'm struck by a thought: Are we the bad guys?
According to the article, what the big tech companies have in common is engineer-led products. It's not the besuited MBA's making product decisions, it's us, the engineers.
And the big tech companies are also united in having evil products. With the possible exception of Shopify and Datadog, every company on that list of "big tech" is doing things that actively harm society.
Cue the "are we the baddies?" meme [0]. If engineers are left to build products by ourselves, do we build things because we can and not because we should?
People who sign up to work on ad services for google or engagement for facebook have already self selected as willing to do morally dubious work in exchange for more money. There's no question that engineers do work that decreases society's utility.
I don't think following incentives makes someone bad though, it's only human nature, and of course there are plenty of people queued up to replace you. The system that creates those incentives might be bad, but it can be impossible to untangle where the incentives stem from.
Pay engineers to optimize a number and they will, that is all you need to do. In this case they told engineers to optimize the number of ads clicked and then the engineers self organized and did the rest.
Maybe evil is just more profitable than good because it externalizes its costs.
Creating a sports car, or paying someone else to do so, is a lot more expensive than just stealing one, if you can get away with it. Robbing a bank is cheaper than building a bank. If you can do this kind of thing at scale, well, to the extent that you only have to spend money on capturing value instead of both creating it and capturing it, you'll be vastly more profitable than the "suckers" you're capturing it from.
Same thing is true of Fecebutt: creating the mountains of data their users go to Fecebutt for (restaurant recommendations, family photos, thoughtful essays, sexy bikini shots, social-network data) took perhaps a trillion person-hours, but Fecebutt gained control over it at a cost of under a trillion dollars. Ballparking, they had to spend less than 5% of the cost of those assets to acquire control over them. Now it can leverage them to organize genocides, rig elections, and extort photos of government IDs and use them to out promiscuous women and closeted gay people—not to mention selling you tens of billions of dollars a year of stuff you don't need through advertising.
It's hard for positive-social-value companies like Shopify and Datadog to compete on salaries with that 20:1 value multiplier. The fact that they manage it at all is not only a testament to the enormous value they're producing but also the enormous destruction of value implicit in Facebook's business model.
From a management perspective, having everything reduced to a process and method is the ideal world, as no true knowledge about the actual work is needed. The weaker then understanding of the work, the stronger the desire to replace uncertainty with process.
However, no process can remove actual randomness (did anyone get the necessary ideas, for example) or uncertainty (is it even feasible, for instance).
That said, in larger - and I'd say mostly non-software companies - changing processes to things that are more inclusive, i.e. tech and the business talking directly to each other does help. And one way to sell that is under an agile/Scrum kind of construct. To me, seeing Scrum then more in non-tech/consultancy makes total sense.
If Skype is an example of Scrum done well, perhaps that explains why Skype has been very unreliable from and end user perspective for 5+ years... the kind of unreliable where more than half of the outgoing calls are met with instant "So-and-so-contact is not available." responses [edit, added: failure responses so quick that you know the contact was never called, and the problem was within your client app or the Skype systems. Different computers in different locations behaved the same.]
Maybe they were missing what I noticed missing from my last Scrum-focused company: view and attention from the higher altitudes.
Scrum makes it easy to break down features and problems into nice little pieces which can be done within one sprint (often in one day). That is a win in many ways, but it doesn't encourage higher level reviews, refactoring, etc. It's the problem of not seeing the forest for the trees.
Of course you can intentionally create epics for these kinds of reviews, but those don't have a good short-term return value for time spent. And so, they get inadvertently or intentionally postponed. Eventually the system is complex and messy, few people understand the system in entirety, and the easiest Scrummy solution is a re-write.
For me, Scrum was a straightjacket, a hundred NOs against my creativity and my attention to broader concerns. There are times when you are implementing a simple ticket, but you keep touching things that need attention... or you see how the entire system has become convoluted, and you see a better way. To do it the right way either means creating new epics and tickets, debating those with product owners who don't understand the value, and generally NOT solving the real problem.
That's not a fault of scrum. If you can't sell the value of refactors on to product, then either you're not making your case or they aren't being reasonable. But you'd have the same problem with any process. We run kanban mostly and we get refactors and tech debt into the backlog pretty regularly. We also reject them pretty regularly when they don't see any viable business cases.
Minor side note: My current employer has been pushing SAFe hard (training for everyone, etc.) recently. As a result (and I do mean "as a result") about 3/4ths of the senior technical people have resigned in the last three months. This is at a "Large, non-tech enterprise" (sort of; it's an engineery chunk of the federal government).
Now back to a personal opinion: Most "software engineering" "methodologies" are intended to appear productive without actually requiring productivity. Or, as I sometimes put it, "software engineering is about making people who are fundamentally not very good at writing code look as if they were producing code." Look at the number of managers, leads, coaches, trainers, and consultants required for every methodology and compare that with the number of such "overhead" positions in, say, open-source projects. (Note: I'm not trying to start a "which is better" discussion; open-source is just a convenient example since their development techniques are transparent, unlike everyone else's.)
Further, many medium- and large-scale initiatives in large, non-tech enterprises fail spectacularly, in spite of using the then-popular methodologies. I would go further and suggest that they all fail at some point, even if they eventually deliver a usable product.
The only solid generalization I can see is that good developers can develop good software with any (or no) methodology; poor developers cannot, and most formal methodologies are built around trying to get them to do so.
I mostly agree. It does depend who set up Jira and why. I’m a developer and have full admin access. I know exactly the kinds of tooling required for smooth integration with the rest of the business. Jira is the one product everyone involved agrees on. (If reluctantly.)
It is a fuck-ton of work to set Jira up so it gets out of the way, but it has all of the customization I need to serve most of everyone’s needs.
If anyone so much as thinks about enforcing workflows on our scrum board, they are going to hear my polite, but firm, rejection of such meddling. It’s our board and we let other people see it so they don’t have to bug us with requests for status updates.
Jira isn’t a hard no for me because I’ve been an admin for it at both this job and my previous one. Let my team manage our stuff, and I’ll happily set up Jira to keep everyone else’s grimy paws off my team.
Seriously, though, having your software enforce too much prescriptive workflow on you is dramatically worse than having it enforce too little. Historically it was common for every software team to write their own bug tracker for this reason.
I have found it to be a great heuristic for companies which focus on process for the sake of it and usually treat engineering as a cost center instead of there reason of being.
One pet peeve of mine: people often say they run a "lightweight" version of some project management philosophy and end up paying the costs of it without the benefits. For example lightweight agile scrum meaning sprints, but no retros and work constantly coming into and out of a sprint.
Is biweekly twice a week or every two weeks? My previous gig had grooming/internal grooming every week. We had stand ups twice a week (biweekly). Almost all of these meetings were relatively worthless imo. They existed to justify a BA's position and "track velocity".
Every two weeks, in our case. Confusing word, sorry. We had two week "sprints". Every day a 15 minute standup before lunch.
We planned what we were going to do the next two weeks. During that time urgent work would also be added and other things wouldn't be finished, but they just moved to the next sprint. Releases were quarterly (customers didn't like frequent releases) so not related to the "sprints".
Some managers described this as "Scrum light", fine with me.
It depends: when I was hired 12 years ago the manager said "we're going to mix agile and waterfall". What he really meant was that there wasn't any methodology at all and I would be left alone on how to organize myself.
I'm still employed there so I guess my methodology (or lack thereof) isn't that bad ;)
As someone who currently works at a Big Tech company, but spent most of his career prior to that at smaller companies, I found this article very confusing.
First of all, I work at Amazon (and posting this under our social media policy, this is just my experience, and I don't speak for the company), and Scrum is pervasive at Amazon. The article is right that many big projects start with a 6 pager, but once teams start executing, many teams run some form of Scrum (Kanban is also popular).
The second thing I found strange in this article was the idea of Scrum being heavy-weight. Early in my career I worked at a small company where our process guru was a big fan of the Unified Process. The point of Scrum is that it is lightweight (without being zero process). There really isn't a lot to it. You sync with your teammates daily, to make sure you're all focused on the most important thing today. You check in every 1-4 weeks by (a) demoing your software to figure out where you are, (b) you do a retrospective to look at what can be done better, (c) you figure out what is the most important thing to be doing the next N weeks. The teams that I've been on that have tried to not do these things either in actuality, did these things, but did them subconsciously, and therefore unintentionally. When I hear people say "Oh, I don't like Scrum because my old team did Scrum and I didn't like how we..." what follows is almost inevitably something that if you asked a professional scrum coach "Should we do that?" the answer would be no.
> Competent, autonomous people need less structure to produce reliable, high-quality output. Big Tech is able to attract, afford and hire these people.
I don't like this way of thinking. Start-ups have lots of competent, autonomous people. Big Tech companies have lots of people early in their careers that are still learning to be autonomous. In fact, when I was at start-ups I often heard the opposite (and equally untrue) claim: that start-ups required people with more autonomy, because the product is new and the team is involved in discovering the product, whereas in larger companies the products are more defined, so people needed less autonomy.
I appreciate the author surveying people and trying to make sense of the results, but in this case I think he just over-simplified a much more complex reality in a way that isn't particularly helpful.
There’s this image that big tech teams build software where everything is automated and has 100% test coverage and all the infrastructure heals itself and all teams are hyper efficient. It’s not really true, the biggest shared practice between big tech is letting the team’s manager pick whatever methodology works best for them.
The big tech version of scrum is just sprints, stand ups and shipping often. There’s still a waterfall process, mounting technical debt, huge backlogs where everything is high priority, and over committing to slipping deadlines.
I think people should think twice before looking at big tech as the place to learn good management practices from.
I think your last point is wrong. Shipping big, complex projects at big tech companies, especially vendors who actually literally ship and can't do live patching to cover up shoddy practices, is actually something _everyone_ in the industry should do because you learn a lot about what it really means to build and release software.
Cloud-delivered CRUD apps are the easy mode of SW development and people who have never done anything else really end up quite stunted IMHO.
I agree with your second point, I often read project management advice and realize it comes from consultants and agencies and small companies and doesn’t really scale for big tech or address issues with large engineering teams.
I actually agree with your first point too, I do think big tech devops and security practices should be copied.
The point I was trying to make is that I don’t think big tech management is actually that good. Big tech is optimizing for staying big (reliability, security, etc.) the famous projects are usually the most uninteresting to engineers cause they ship a handful of new features a year and you spend most of your time working on bug fixes, infrastructure updates and security patches so those projects end up having the most churn in engineers and management. Because of that churn these teams tend to adopt the practices most suited to oversight (stand ups, sprints, retrospectives, etc.) instead of the ones best fit for improving the customer experience (grooming the backlog, collecting user feedback often, etc.). So that’s why I say we shouldn’t be using big tech management practices as an example imo, cause big tech still hasn’t decided on a solution to the problem of slipping deadlines, exploding backlogs and insurmountable technical debt.
Just to be the old grouchy guy, I can't say that any of the modern processes sound like any fun. 90% of the time I've run into them it was basically a way for external consultants to make a pile of cash (and for an internal champion to work without doing work), the other 10% it slowed down development to a series of small changes.
Perhaps it's just a difference in the scope and type of projects (boutique hardware vs. large scale online stuff).
Looking back at old codebases that I kept around, we managed to build quite complicated largish systems in a reasonable amount of time using straight waterfall/ad hoc approaches. Simple source control. Simple spreadsheets for bug lists.
Admittedly it's hard to avoid attaching particular people to particular blocks of a system, so perhaps making development capable of absorbing random Engineering Resource Units (humans) is the point of all this.
..or maybe it's just that earlier generations of programmers were stupid. I can accept that.
The software process stuff I've found most valuable is really the XP/Joel-Test stuff, not the Scrum stuff:
· nightly builds or continuous integration;
· pair programming or code review;
· automated builds;
· comprehensive test suites or test-first programming;
· fixing bugs before adding features;
· source code version tracking software;
· keeping a list of bugs and planned features instead of not keeping one;
· prioritizing planned features instead of not prioritizing them;
· merciless refactoring to keep the design simple;
· DRY, as a criterion for what "simple" means;
· "YAGNI" (not writing code to provide functionality that I'm not implementing right now);
· a sustainable pace (no death marches);
· exploratory "spikes" to explore unknown features;
· taking regular breaks;
· informal usability tests with prototypes;
· standup meetings;
· frequent releases;
· quiet working conditions; and
· retrospectives.
I'd add, though these usually go without saying these days:
· high-level languages;
· interactive development environments, as opposed to the batch-mode approach where you submit a batch job and get back your program output an hour or a week later;
· enough testing hardware that you never have to wait for a testing machine to become available in order to get your work done;
· using available software libraries instead of not using them.
I'm pretty confident that each and every one of these helps me build better software faster, though a few of them may vary somewhat depending on circumstances—I know a guy who can concentrate to write code better in a noisy café than in a private office, frequent releases are at best minimally useful for pacemaker firmware, and you couldn't have written qmail using existing libraries.
Also, though, I'm guessing that the old codebases you're looking at with simple spreadsheets for bug lists practiced "working software over comprehensive documentation", "customer collaboration over contract negotiation", "responding to change over following a plan", and especially "individuals and interactions over processes and tools". And those are the core aspects of agile development (though not the Fake Agile we so often see).
Some of these are things earlier generations of programmers could only rarely do. If you're working at a company that values processes and tools over individuals and interactions, there's not that much you can do to fix that if you couldn't start your own company; https://news.ycombinator.com/item?id=28670326 suggests one reason this is so widespread. Similarly for fixing bugs before adding features, using an interactive computer, having adequate testing hardware, and having quiet working conditions. Source control systems, automated builds, and automated test suites provided substantial benefits to teams that adopted them, but many didn't. That doesn't mean they weren't valuable at the time, just that many people did without.
By contrast, I'm less convinced of the value of private offices, written spec documents, schedules, dedicated low-paid testers, scrum masters, planning poker, splitting and merging product backlog items, quantitative task estimation, and even coding interviews.
> Many teams work on main branches, get quick feedback from CI/CD systems and can immediately share functionality which they are working on with other team members.
I occasionally see statements like this floating around - does he mean:
* Devs make their changes locally, commit and push directly to main, and then the CI/CD either notifies them that tests failed or deploys to prod
or
* Devs make their changes locally, commit and push on a branch, and rapidly (ideally automatically) merge to main if tests pass. This happens on a cycle of minimum coherent set of code changes, not a whole feature at a time or anything like that.
The first seems like it would be very frustrating if code with failing tests if pushed with any frequency, but seems to be literally what the phase "work on main" means. I also don't really see a drawback to the second one in comparison.
Every commit is code reviewed separately and merged to master after review + automated test gatekeeping as necessary. Additional testing may occur downstream post-merge which may cause your commit to rolled back automatically. Time to deployment into production depends on how those deployment pipelines are built, could be minutes to days before it lands.
It's very unusual to merge more than one commit to master at a time.
You can think of it as a branch and merge approach, but the set of changes is small - a 100-line diff would be a large change (automated changes and deleting unused code excepted). The process and tooling optimizes for small changes. Refactoring is done incrementally. Big changes are gated with flags so they can be built incrementally and rolled out incrementally and turned off at the first sign of trouble.
I can’t speak for all companies but at the place I worked at, there was a monorepo. You’d create a small, self contained change off the main branch and put it up for review. All such proposed changes would have tests run on them, so the reviewer could get a sense of the quality of changes. “Looks good, please fix that test” was a comment I’ve seen more than once. After approval, it’ll land on main and be in prod within an hour or two.
For larger features, these changes could be stacked in a list. After each change in the list had been approved, they’d be landed on main as an all or nothing.
> The first seems like it would be very frustrating if code with failing tests if pushed with any frequency, but seems to be literally what the phase "work on main" means.
Yep, that's the point - it's to discourage devs from checking in failing code (because then they'll be swarmed by annoyed coworkers). It's fairly common to see developer A make a breaking change to some low-level library and then see developer B push a rollback of A's changes a few minutes later (with or without A's permission).
At Google your tests are automatically run beforehand, but not all tests of entire Google are run for your commit beforehand. Sometimes you break others code with your change.
Edit: Just to clarify, it isn't ok to break downstream projects, if you do the change gets rolled back almost immediately. And you can run all tests for the change before submitting, but it isn't default since it is expensive for many core projects to run so many tests.
This seems to be advocating the second workflow, i.e. pushing on a short lived branch then merging back in (their intermediate setup seems impractical technically for most setups - how do you run standard tests without committing your code, unless you do everything locally?)
> This seems to be advocating the second workflow, i.e. pushing on a short lived branch then merging back in
No, in trunk-based development, you push to the trunk, aka master.
The difference is perhaps not that big; with TBD, you have a branch, but only on your machine, which you merge to master before pushing, whereas with the second workflow, you push your master, then merge somehow.
> their intermediate setup seems impractical technically for most setups - how do you run standard tests without committing your code, unless you do everything locally?
I don't understand this. Of course you can and should run all tests locally.
Even if you really wanted to have it be trunk-only, I don't know why you wouldn't make everyone push to a feature branch, then have automation for running tests and merging after they pass. Plus, where's the code review in this process? Even if your team is all amazing developers somehow, it's insecure to have a standard process where developers are pushing unreviewed code directly to production.
Fundamentally this all seems like a misreading of trunk-based development, though. The whole idea was to get away from old Subversion-style branching, where there would be long-lived branches running for sometimes several month. If you were a company that moved from svn -> git, trunk-based development was a way of communicating that branches were cheap and disposable, and merging was quick and easy.
I'm questioning the logic of having a workflow with a not-uncommon case of "this is on fire now everyone stop what they're doing and help me fix it" which is exactly what you'd have here.
One of the premises is that the minimum level of competence of the team should be above average, which is not feasible everywhere.
And
"The short-lived feature branch should only last a day or two and never diverge from the trunk enough so that a merge back is problematic. After the seal of approval from code reviewers and CI daemons, it should be merged back into the trunk. It should be deleted, as proof of convergence. The developer in question may then go ahead and make the next short-lived feature branch for the next story/task they’re doing"
https://trunkbaseddevelopment.com/youre-doing-it-wrong/#dura...
I worked on a Tensorflow feature in 2018. There was no branching involved. Fork the main repo, build the feature on your child repo, make sure all unit tests pass and then send a pull request to the main/master of the parent repo. Typically someone reviews the code, you incorporate code review comments, pass all unit tests, CI-CD pass, PR approved. That's it. Your code is in the next production cycle.
So yes, you work locally on a fork of the repo but not the main/master repo. I'd surprised if all devs have access to push to main/master.
I think having branches leads to a lot of merging issues. Because technically, the branch will then be used by multiple devs. So it's like dealing with multiple masters, kind of.
Personally, I prefer the TF approach. Oh I forgot to mention, the release tags are branches I think, perhaps read only branches.
This is just branching, except your branch is in a different repo. You just call the branch downstream/master and your local master, instead of the more common origin/yourbranch and your local yourbranch.
For the rest the merging issues are identical. You can still have a merge conflict between downstream/master and origin/master if someone merged something conflicting into origin/master since the fork happened.
What you say is certainly true, but there are still meaningful differences: named branches give you the option to push them before merging as an informal remote backup (good) switch to a different named branch (neutral) and let the old one grow stale (very bad)
How would you describe the difference between branches and forks? Functionally I can create a new branch, do my changes, get tests passing and PR from there, or I can fork master, do my changes, get tests passing and PR.
Yes technically this is a branch, but like a frictionless branch. You don't have to name a branch for instance. You don't have to worry about branch deletion, even though now this is all automated.
Ah right, generally I quite like having branch names because I can name the brach aligned to the jira ticket that it's associated with, I'm not a big fan of jira, but atleast it documents the feature request etc. And also you can track multiple different features in flight at once (I know it's not ideal, but sometimes you're doing several different things that need to be merged in order etc)
One of the bests parts about working at a faang is we don't do scrum or stand ups. Everyone shares timelines ahead of a project, and they run with the execution, flag if there are any blockers or need help. Saves so much time without the extra overhead.
My theory is that in tech companies top management usually understands how tech development works and have respect for their workers.
At my company top management often comes from a sales or medical background and you can clearly see that in how they treat sales people and medical people. They get promoted more and receive way more attention from management because they understand each other.
IT and software are a big, scary mystery to management so they are prone to take advice from snake oil salespeople, consultants, or middle managers who tell them what they want to hear.
I think in general I think you are better off in companies where the output of your work is the product and top management understands and respects your work. Better to be viewed as someone who contributes to the profits of the company and not just as a cost center.
Same here. Worked all my career on medical/biotech as software product and project manager. I‘ve even heard management telling me „this is just code, should be copy paste“ and „why is this taking so long that should be easy“… and if things of course don’t go as planned they layoff and hire Accenture.
My role (as I see it) is to keep devs motivated, translate this mystery of development for them and present some kind of roadmap. As much as I would love to only work with empowered teams, the reality is that with these big nontech companies are not ready to change anytime soon.
I don't quite understand the "lack of Scrum in large companies" when they all seem to Plan-Ship-Build. That is pretty much the high level description of Scrum isn't it?
Sure Scrum has some ceremonies, but these are basically planning and retrospective for improvements, so again, not really any different.
At the end of the day, most of the ways I have worked end up with Tickets/PBIs/Stories and you work through them. The only major difference is how much in-advance these are specified and to what detail.
I’ve worked at a couple big tech companies and one scrum. Notable differences, much less ceremonies, planning is also more up front with deviation allowed but not needing additional planning, and ticketing being optional thing that varies by team.
I do not usually track my current task with a ticket or story. I have once week meeting with my team to discuss what I did last week and what I’ll work on next week. There is no 2 week sprint/constant stand ups/individual ticket estimates/retro. A typical plan is for a project that will take a month on short side or 2 quarters on long side. Deadlines tend to be soft with it culturally accepted that most deadlines are rough estimates and a week or two so deviation being fine. Process feels very light for me and I’m pretty happy with that.
Of the weekly "Scrum sucks" posts we see on HN, this was by far the most articulate and well-reasoned. I've spent most of my career in consulting or kitchen sink teams. Having Scrum or a Scrum-like process is simply mandatory because we are working with disparate stakeholders against constrained budgets with a team who are not dedicated to the domain we're working in. Getting a high degree of specificity into requirements in the smallest increments is life-saving. Over reliance on rituals is typically only necessary for a really immature team (which I have been part of in the past) but the core elements of backlog management are really valuable. Not just for developer efficiency, but for visibility on progress and priorities.
All that being said, I can easily see how a dedicated team with a very clear product strategy iterating on a hugely successful business model can just roll along without a lot of oversight. I'd still be a bit surprised that anyone working on a customer-facing product isn't running their design decisions through product experts. Idk what that world is like, but I've spent a lot of energy duking out product debates where nobody is really sure how a feature should work or it's even beneficial and developers usually have the worst instincts.
So the key takeaway here is that the Plan, build(iterate), ship cycle works very well. Most successful teams stick to this. What is not clear to me is how an abrupt new feature with an urgent deadline fits into all this ? "We'll fit this into the next release cycle" is not an option, let's say.
So how do you fit in this new requirement ? You are already in the middle of a feature built on day 3 or 4. Devs are working on their forks and have uncommitted code. Now you have a new feature that has to be given priority, assuming there are no dependencies on the existing cycle. Do you just keep a copy of your old work, and start working on this new stuff ? Feels like a mess. Any better approaches ?
In my experience, there is no good solution to this problem- when feature X comes along and it is decided that the ongoing work on A,B,C need to be paused because X is that important, there really is no way to quickly context switch back to A,B,C after X is finished.
The best thing you can do, in my experience, is make sure that the devs are fully bought in on why X is so much more important than A,B,C. Ideally, you want to present it to them as 'X enables us to get this contract/solve an uptime issue/whatever' and then let them decide, as a group, that maybe we continue on B while the two working on A and C instead focus on X.
> What is not clear to me is how an abrupt new feature with an urgent deadline fits into all this ? "We'll fit this into the next release cycle" is not an option, let's say.
Maybe the larger, successful tech companies avoid doing this. I believe that most of them define work at a quarterly level. So a team is unlikely to have the sudden changes in direction.
They are also unlikely to be resource-constrained, so when the odd high-priority feature emerges from the ether, they can put together a new team to tackle it.
Smaller companies need the discipline to know not to suddenly change everyone's priority, and instead, work emergent features into the current backlog at a pace appropriate for the size and scope of their teams.
What might an "urgent new feature" consist of? Well...
The Swiss Franc is no longer pegged to the Euro, so our trading software needs to track a separate currency, and also we need to stop trading EUR/CHF forex pairs as if they were still pegged; we lost $2 million before I turned the system off this morning. The new version of IOS doesn't support the old SNMP MIB, and our customer has already installed it. The vid.me domain has been bought by a porn spammer and now we have hardcore porn ads on all our news articles.
YouTube revved their AJAX and our video metadata scraper stopped working. We're being hit by an 0.1Mbps DoS because this one page takes a whole CPU second to render. The new satellite downlink site came online six months earlier than expected, so our comms software needs that multisite feature next week instead of in six months, or we're paying a shitload of rent for nothing. Modi demonetized large bills last night, so we need to stop accepting them at our ATMs. Apple told us yesterday they won't approve our update to the iOS app unless we add in-application purchases (and disable every other payment method).
We signed a contract to provide security for a new film, but they're filming onsite at an Air Force Base, so we need a field in the database for which security guards are cleared to go on the base, and filming starts on Monday. Turkey just ran all their ISPs through CGNAT, which means we need to fix our NAT traversal strategy so it works through two layers of NAT, or we lose half our Turkish users. Russia is fining us $100,000 a day until we move all the data about Russian users onto our Russian server.
Our popular webcam software just doesn't work with this popular new webcam. We've identified a new signal in the market that we think will give us 0.7¢ per trade, but someone else will probably exploit it within the next two weeks, and then it won't be profitable anymore. It looks like Berkshire Hathaway is going to break US$671088.64 per share this month, and we need to be able to handle that. Apple just announced that in iOS 11 the default video codec is HEVC, so unless we tell our users how to reconfigure the Camera app (and they reshoot their videos) we need to be able to handle HEVC uploads before people upgrade their phones to iOS 11.
NCMEC just gave us the new version of their damn fuzzy hashing algorithm; they'll start giving us hashes in that format next month, and if we don't support it by then, we're kiddy porn purveyors. Oh, Argentina canceled daylight savings time with 11 days of warning, so all those calendar entries our users scheduled two weeks from now, they now have the wrong UTC time and need to be fixed, plus we need to roll a timezone update. We finally signed the deal with Nicki Minaj to make her a character in the game we were about to ship, but we can't slip the ship date past Black Friday or our sales will take a huge hit.
(Names and details have been changed to protect the guilty. Some events haven't happened yet.)
I have no idea what project you work on, but that sounds like 30 different teams each experiencing:
1. Rare actual urgent issues
2. Shit leadership adding last minute requirements (that no scrum or waterfall or agile whatever would help with)
3. A bunch of config changes (I worked on payments systems in the past. We had some contractors who would basically mess with config files all day as countries changed laws, etc).
4. Stuff that only quant firms care about
Regardless, you didn't address the second part of my comment: What does the fact that we'd have to stop other work have to do with anything? That'd be the case for any workflow structure.
Make sure the features you're working on can get committed to the main branch in small, incremental units. Dark deploys/feature flags help avoid big long-lived branches that go stale and have to be kept in sync with main. So in your scenario, the devs finish off the current tasks they're working on, merge them, and move to the new work. You'll still have the pain of some feature being in an incomplete state for a while, but it's not affecting your users (who can't see it yet) and there's no half-abandoned branch slowly rotting.
hm - I'm at a startup with plan/release/build. The truth is...there are almost no abrupt new features. If something is truly a new feature - it's gonna move into planning. We've had 1 instance in my 8 months on the team with an abrupt-ish full-on feature (that I know of at least). I worked on it - there was a lot of communication with me about the scope/timeline/expectations. The feature fit into an offering we have at a beta stage - so it helped us expand something we were working on broadly anyways and was in service of winning a particularly good logo/customer. Again - this is super rare for us, but broadly - we simply communicate priority and have alignment about working on the most important things first. Generally, someone is picking up the feature that's coming off some piece of work or is working on something low stakes. Other abrupt stuff comes up - generally if it's limiting a customer's ability to utilize our product - we prioritize it.
I think you want to have a culture where abrupt new features are incredibly rare.
Version control will let you set the work aside and resume it later. You might have to burn a few cycles merging your old code up to date, but it won't be lost.
I haven't really seen a better option. Urgent work is disruptive to schedule by definition.
I think the core observation that teams are best allowed to adapt to their own situation is a good one. One thing I want to take on though is this:
> “Kitchen sink teams” which have everything thrown at them, typically find that managing stakeholders with a heavyweight process like Scrum is a win. Stakeholders are educated to understand that an ongoing sprint cannot be interrupted and that new feature requests need to be groomed.
I have repeatedly see teams that run like this fail. They're happy in their own right, they've got their process, they get what is in front of them done, but the process makes them entirely unable (actually more often just unwilling) to respond to urgent issues or adapt to a changing environment.
It's absolutely great for a team that has a large number of external stakeholders with diverse needs to be able to point at some process and say "Oh well here's the process we follow, here's what you need to do". But in reality what they're saying is "We're entirely unable to respond to the actual requirements of our stakeholders, so instead we've decided there are only a subset of requirements we're going to fulfill" and anything outside those requirements you just have to work around the team. It very often also works to provide barriers between the stakeholder and the engineer, so by the time they tackle they're either doing the wrong thing, or it's no longer needed or they fail to actually deliver it to the stakeholder. It's a great system to build a well functioning team that entirely fails to deliver what the larger organisation needs. /rant over I guess
That's where short cycles come into play. If something is urgent for business (and often everything is "urgent") then it can fall in the next cycle which is just 14 days away. It's rare that something is so urgent that it should disrupt your entire dev team flow and focus.
Also what happens is you end up with a never ending dev cycle where business expedites an item making the cycle a little longer, business sees the cycle is longer and thus adds in more items to be included further lengething the cycle... I think you get where this is going.
At my startup company we have suffered from just this in past, where we would have never ending dev cycles while our maintenance back logs filled up and never got worked on properly.
At our company, We believe that the way you get to the objectives and result is immaterial. Some teams that are very IC driven, will not care about the process and but will be jet focussed on result. Some teams need structure, some don't. Enforcing same process on all the teams is kind of shackling, what ends up happening is that teams do process for the sake of it. Our teams have occasional alignment calls, team and individual OKRs, company objectives are the key indicators for the manager to see if the we are on track or not.
Nah, it was created to add a new industry on top of IT. Everyone is doing Agil.. I mean Scrum. So everyone need to be certified, to follow workshop, to follow the "innovation", to buy the associated toolings and so on.
No, I want to be able to represent myself as authoritatively as the guy who proclaims to be “scrum certified,” as though scrum is so valuable as to have a certification.
> ...Skype in 2012, the company had gone all-in on Scrum. All engineers and product people were sent to best-in-class Scrum training, facilitated by one of the Agile manifesto’s founders.
We trade the story of one Agile Founder wanting to kill his monster baby, but the others have been making bank.
I'm well aware of when/how Scrum was invented, hence "tinfoil hat".
The point was that whatever the origins and intent, Scrum benefits Big Tech (who largely avoid Scrum as per OP) by hobbling smaller competitors who adopt it.
I somehow feel something important is not being discussed, and that is competency. Perhaps in Silicon Valley it is normal that engineers take ownership and are smart enough to figure out what is valuable and what is not, but I can assure you that in an average Fortune 500 company this is not the case. In my experience a lot of engineers in these companies don't care about engaging without customers, like to complain about a management and take little responsibility.
You need to have rules (however dumb you might find them), you even need babysitters. There's no going around that. Someone making 1/4h of what a SV engineer makes is highly likely to be less competent.
I worked at a big tech for many years and my team never managed to get Scrum working properly. Every year my team commits a delivering certain product/features at a very specific date (some sort of launch event), so we have to know very early in the year what's all the work required and report periodically whether project is on track. The deadlines are also always on the tighter end. The flexibility of Scrum becomes an issue in that case, because not everyone can deliver the same tasks at the same speed and without careful planning you can easily miss important deadlines.
You cannot get management buy-in to Scrum or most agile processes unless they can accept the agile manifesto principles.
The whole idea that you can be agile while having delivery dates for things several months in advance is a nonsense. Agile is about doing what is important now, not doing what we thought was important a few months ago.
The best you could probably hope for is Kanban or the never-ending backlog but some people struggle when there is no finish line, just an endless sea of work!
>The whole idea that you can be agile while having delivery dates for things several months in advance is a nonsense
If Agile truly rejects even the possibility of an estimation (something I'm sure many proponents will disagree with), that would be a problem with Agile and not a problem with the idea of estimation. Companies will de facto pivot away to methods that will give an estimation, even if they're 'bad Agile' and even if software engineers are unhappy about it.
Because there are more useful things to optimize for than efficiency of software delivery or what software engineers like. People really need to know when the works on their street will be done, companies and countries need to plan ahead, and so on.
We just delivered a multi-year project under Scrum (edit: point of clarification, we did not have a due date). You still have a guiding vision and some idea of the major milestones you need to reach, you still define what the next set of milestones are. You will want several sprints worth of work lined up ready to be more concretely planned. You just do concrete planning for the next sprint or so and work to minimize disruption of that sprint. Learnings being incorporated and plans updated as needed. Work that will yield more understanding should be prioritized.
Not sure where the idea of "only look one sprint into the future thing" came from.
"The whole idea that you can be agile while having delivery dates for things several months in advance is a nonsense. Agile is about doing what is important now, not doing what we thought was important a few months ago."
I agree with you - i dont think its worth trying to set a date a year out in stone. but you have to do exactly what you say. keep the trajectory in your mind and update it with new information. and and some point turn down the rate of change so you can get some cook time.
maybe I reacted too strongly - but I've certainly been in shops where I try to talk about overall project arc and am firmly told to keep my gaze fixed at exactly two weeks.
If you have a set date and a set scope you can't be agile. Agile needs to be ale to radically change scope or even cancel projects outright. You can get lucky sometimes that a feature takes less or about the same time as you have and make things look agile.
The flexibility of scrum is being able to say what is done represents a working though not fully featured product. You can generally send red flags a lot faster in scrum, but you still need management that believes and understands those flags.
> If you have a set date and a set scope you can't be agile. Agile needs to be ale to radically change scope or even cancel projects outright.
The thing I think that fails to be recognized about Agile (though there is a lot adjacent to it, without outright calling it out, in the central documents) too often is that Agile has fundamental tension with the project-oriented, and even to a lesser extend product-orient d, mindset, and really calls for a mindset in which technology is an integral component of continuous end-to-end business process improvement. It can be adapted to product- or project-oriented environments, but only with compromise.
> The flexibility of scrum is being able to say what is done represents a working though not fully featured product.
The flexibility of Agile in general comes in rejecting products as a goal to themselves, and recognizing that technology is part of a broader system of delivering value and any improvement in value delivery is worthwhile according to the value delivered compared to the cost of delivering it.
This item stands out as one of the most important aspects that is either unknown or ignored in companies. The amount of information a person has access to drastically alters their ability to make informed decisions.
"Exposure to the business and to business metrics. Engineers are encouraged to interact with the rest of the business and build relationships with non-engineers. In contrast, traditional companies often make it impossible for developers to interact with the rest of the business."
> It measured a Net Promoter Score (NPS) of -83. This is staggeringly low, and means that 83% of engineers would advise against JIRA
No, that's not at all what it means. Another person who does not understand the bullshit that is NPS. NPS treats people rating at the lowest of the scale just like people rating at the middle of the scale, which is patently ridiculous.
It's not exactly what an NPS score of -83 means, but it's close enough and an NPS score of -83 is sufficiently awful that it should be considered a crisis for any business that does not have strong customer lock in. Actually a few promoters in the sample would potentially imply that more than 83% of engineers would advise against JIRA.
Of course NPS scores overlook some nuances. It's a single estimate of a complicated space. The imprecision is probably to its advantage. The goal is not to be perfect (no single number could do that), but to provide a quick estimate of quality with a single question (nobody wants to fill out a 3 page survey, and the executive team is only going to sit and listen for a single statistic).
As an exercise, consider how you chose a restaurant based on a standard Yelp 1-5 scale: you might find that you are using a similar methodology while reading product/service review. Ratings of 1-3 are considered bad reviews, 4s are decent and 5s are good. American culture as a strong bias for positive affect so a 3 is considered to be unfavorable. Additionally, there is a lot of variance in terms of how people get upset and express their anger. Someone may rate something as a 1 because they are more expressive of their angry, while another person rates it as a 3, but neither is really happy with the service. Try coming up with a better metric based on yelp reviews for restaurants you are familiar with and you might find that NPS does a reasonable approximation of what you have discovered.
NPS is a marketing tool. I agree it's misused in this case, but the people at the extreme ends of the scale DO matter. They are the ones who will passionately voice their opinions, write reviews, do blogs posts, etc. The people in the middle of the scale can't be bothered with doing anything about their opinions.
The way we do things at Gordian is similar in a lot of way to the Big Tech methods but it stands out in a pretty unique way IMO.
The approach is described here https://www.gordiansoftware.com/news/two-weeks-too-slow and in practice it’s even better than anything I was hoping for. I can legitimately say this is the most productive environment I’ve ever seen.
I don't get Jira hate. It may be clunky, slow and overcomplicated but really it's just system to document stuff. If you don't document stuff on a sufficiently big project (I'd day >10 engineers, and especially at >100) then it's a recipe for a disaster.
As for solutions to document stuff, specifically issues, then Jira seems to be ok and not lacking in any area, maybe not a leader but not a worst thing to use too.
> I don't get Jira hate. It may be clunky, slow and overcomplicated
Ding ding ding!
Seriously though, I think a lot of issues with Jira stem from poor implementations that cause it to perform terribly, which impedes an engineer's ability to quickly get in and out of Jira. For example, the on-prem Jira instance for one of my clients would take upwards of five minutes to render the board, and another 30+ seconds to render the ticket page. Adding comments was another frustrating experience. I'm sure the OOTB experience with modern Jira is probably fine, but it seems that hardly anyone uses Jira without some resource-intensive customization.
What is needed is registrars keeping track of documents and writing summaries etc. I blame the 'savings' on secretaries for many of the modern corporate worlds problems. A tell tail of that the top brass them-self don't believe secretaries are not needed, is that they them-self have them. It is just engineering that can't have them - pushing any bigger engineering org. into chaos.
Scrum is merely a batch oriented process. In (process) control theory there is over 100 years of study of batch, continuous flow (e.g. Kanban or Lean), and other types of processes.
Each has benefits and tradeoffs. Like all tools the proper process type for your project will be context sensitive.
For an industry so full of smart, supposedly logical, people ours is still quite immature and tribal in its management theory.
The initial comparison - WhatsApp (18 employees) vs Skype (hundreds? thousands) is not a comparison of Project Management systems but a great example of the Clay Christensen Innovators Dilemma - Skype got killed by a mobile only, E2E encrypted competitor.
I think the big takeaway is "dont worry about your project managment system if you are playing the wrong game."
Not surprising that scrum works best for consultancies -- Scrum has just as much to say for how engineers should behave as it has to say about the people paying for the project should behave, and how to pretend to be that person when they seemingly abandon their duties.
The sprint is also a product of its time. A methodology for shipping new features on a monthly basis is great when your competitor is shipping Excel on an annual basis. But the shorter the turnaround time for a feature, the less use there is in planning timelines.
This is why the big tech companies don't have a unified process -- the process that works best for shipping operating systems is different than the one that works for shipping a mobile app, is different than the one that works for shipping a webapp. Any program office that tries to smooth out these wrinkles is hazardous to their wealth.
Kudos to the author for gathering some data and sharing their conclusions. I encourage you all to take a look at the survey responses though. I know Scrum and Agile bashing is fun and all, but we're drawing some pretty big conclusions from just over 100 responses. And while the focus of this seems to be pitting "big tech" against others.. with the assumption that we're talking about FAANG.. there are 16 responses from public tech companies (if I'm understanding this spreadsheet correctly).
"We went from shipping the flagship Windows app once-a-quarter at best, to monthly shipping"
Here's some professional advice: The idea that releasing more often is a good idea is total rubbish. Eric S. Raymond said it originally ("release early, release often") about open-source. People have applied it to software in general because of survivorship bias. Web devs tried it, and surprise, web dev can get away with shipping an update multiple times a day. Your users probably won't even notice.
But what if your program isn't a webapp? Updates aren't so seamless. If you update Skype 3 times a day, your users will be prompted to update Skype app 3 times a day. Do you really think that's a good idea?
XP was developed in the Chrysler Comprehensive Compensation project. The project began in 01993 and brought in Kent Beck to reboot it in 01996, at which point they still hadn't gone live; they went live in 01997 and continued work on the project until 02000. Frequent releases are one of the cornerstones of XP — every 2 weeks, I think, on C3. You can read about it in https://martinfowler.com/bliki/C3.html and https://wiki.c2.com/?ChryslerComprehensiveCompensation.
Releasing often isn't always a good idea, but it isn't necessary for every user to upgrade to every release. Here are some advantages of releasing every iteration (once or twice a month):
1. You can never have the software in an uncompilable or unshippable state for an entire iteration, much less three months at a time, so the only time you don't know how long a showstopper bug will take to fix is immediately after you find it, and you're never "integrating, don't know how long this will take," because frequent releases force continuous integration. You never slip a ship date, at least not by more than a week or so; you just cut scope. Slipping can be enormously painful and sometimes fatal to a business.
2. A hacker costs a business US$200k a year. A team of ten hackers costs US$38000 per week. If you release once a quarter, then, on average, at any given moment, you have six weeks of work that you've paid for that haven't reached users and are thus not yet producing any value; that's about a quarter of a million dollars of "inventory"—having a lot of inventory might sound great if you're a survivalist or an RPG player, but for a business, "inventory" is wasted money. If you can cut that to monthly, you only average two weeks of inventory, so you've saved US$170k, almost the cost of a person-year of work.
3. When you ship software to users, they react, which is an opportunity to learn from them about what they value and what they don't; this both gives you ideas for new features and gives you more information about the value of features you'd already thought of. This is true even if only some of your users install the new release. If you ship once a quarter, you get four of these round trips per year; if you ship once a month, you get twelve. So you can learn a lot more from your users. Evidently that wasn't enough to get Skype on track.
4. If you're a business, the business environment changes as you're developing software: for example, your competitors add new features, systems you interoperate with change their behavior in ways that break your software, and new libraries become available. To remain profitable and in particular to remain better than the competition, you must observe these changes, orient yourself with respect to them, decide what to do, and then act. In competitive environments, the competitor who can go through this "OODA" loop faster will almost invariably win. Shipping features that embody the right decisions for this month instead of the right decisions for three months ago can be a decisive advantage—though I think this is what WhatsApp was better at.
5. When your sales guy meets with a potential client on October 10 and finds out about a missing feature that's a showstopper for that client, and the October 28 release now has that feature, it gives you totally different chances of closing the deal than if the January 10 release has the feature. Even if they don't buy then, you've made a totally different impression, and you're more likely to make a sale later or get a lead on another potential client from them.
6. More generally, frequent releases make progress and quality legible to people outside the programming team, circumventing the "thermocline of truth" problem named in https://brucefwebster.com/2008/04/15/the-wetware-crisis-the-.... Having five features "80% done" after four months could mean you'll deliver them in another month, or that you'll deliver them in another 8 months. By contrast, the difference between delivering four of those features in four monthly releases, and delivering one of them, is a difference observable to users, management, sales, and customers. Sometime it's even helpful for keeping the people inside the development team from fooling themselves, despite being below the thermocline of truth.
I'm writing this based on the experience of utterly smashing two established competitors at AirWave, where upgrading one of our clients always involved a field engineer, often doing a site visit. We weren't writing open-source software, either.
> Scrum got in the way of shipping on a daily basis. The whole idea of Scrum revolves around Sprints, of committing to tasks at the beginning of the sprint, working on these during the sprint, and demoing what we did at the end.
What if each team member came up with a few tickets to work on during let's say a 2 week sprint and then these tickets were shipped as they were finished resulting in releasing new code potentially every day or couple of days?
I've worked with a bunch of companies (contract work) and very rarely did they stick with 4-6 week sprints but only shipped everything at once in the end. That would be really dangerous. Almost always things were shipped as they were completed and reviewed.
You forget that there's a big administrative load added to a two week scrum (planning, daily, daily...), that you need to test and validate, potentially validate the work with QA and do a demonstration at the end. I've never seen a two week sprint working as intended.
Especially if you factor the interruption for fix, new feature discussion, task that are more complex than expected and so on.
If you add a manager in the loop, pressuring the team member to agree to unrealistic deadline, Scrum is pretty much the best way to destroy everyone involved.
I think most people I see doing scrum these days, do a story, demo a story to owner, release story. Move onto next story. Rarely do they release at the end.
The review at the end is for all stakeholders to be able to catch up on what's being going on the last 2 weeks.
> You forget that there's a big administrative load added to a two week scrum
It wasn't forgotten. Maybe the places I've done contract work for didn't follow scrum to a T. There was no daily planning. It was meeting once for 2 hours every 2 weeks to figure out what to work on and then a few small teams self regulating themselves asynchronously until things were done. This included doing the work, updating the ticket, having at least 1 person review / test the PR and it getting released. Usually there was a 1-3 day turnaround time from the PR being set to review to it being live on the site. Could be longer if the PR needed a lot of changes from things caught in the review process but the ball was always moving.
It never killed a single developer's productivity to wait on a review because every developer had a few tickets to work on during that 2 week process so they could jump to the next one or review another person's PR in the mean time.
There was no manager component once the tickets were chosen for the sprint (which was often a manager + developer group effort). Code could go from a ticket to production with no administrative bottlenecks.
I think the biggest misunderstanding that always occurs when talking about Scrum and/or Agile is when people from radically different industries meet. From the perspective of my previous team, 1-3 days from PR to prod would be indeed quite long. For such a team, "two weeks per sprint" can seem like an eternity and will only slow things down.
On the other hand, one of my best friends works for the largest insurer in the company. They do about one release per six months and working in two week cycles would be an unimaginable speedup for them. It frequently takes more than a month to get word back from the regulators that their proposed code changes have been approved at all.
Both of the groups ("scrum slows you down"/"scrum is uncomfortably fast") have trouble imagining that the problems of the other group can even exist.
What would you do to go from PR to prod faster when you want at least 1 human who didn't open the PR to manually review each PR in a 10-20 developer remote team?
1 day doesn't seem too bad, often times it can be 2-6 hours. It really depends on what lines up and how big the PR is.
A code review for a smallish change shouldn't take more time than sending a few chat messages. You send them the code review, they look through the 20 lines in the code review tool, note that the changes corresponds to the change description, see that you added tests for it and that those tests passed, they ask you to clarify a variable name, you update the code and send it again a few minutes later, they review the new change and see that the name was properly changed and tests still pass and press accept.
This is how most of the code reviews went for me at Google. If you do a bigger change it will take more time, but I see no reason why smaller should take longer than that. After all you are just sending small bits of text back and forth, it really isn't more complicated than a few chat messages.
The delay isn't always around the review itself taking long. It's having a dev working on their own tickets and then having an opportunity to check in and review someone elses code. You could end up getting a review on your code in 10 minutes or 5 hours based on what's going on for everyone at the time you open the PR. It might even take until the next day and for a bigger PR it could involve a few more eyeballs and time commitment for the review itself (outliers but they do happen).
s/Scrum/Kanban in most of these critical responses and it would still be pretty accurate of the state of most applications of it.
I can't help but feel that people say Kanban because Scrum is so uncool. You can see that from the roasting Scrum's getting here. With these visceral reactions, I can see why people are cautious about saying they do Scrum.
As for Kanban, most teams are no more doing Kanban than they are Scrum. That is, they're probably just following Jira's process. I've asked candidates in interviews what they mean by Kanban (e.g. if mentioned on their CV) and the common answer is, "the process we use in Jira".
Scrum is one of the worst things ever to happen to Software Developers. The embodiment of metric rigging, and Developer disenfranchisement. A dream for a certain type of manager. I changed career not to have to use it.
Can you imagine Linus, John Cormack or Brian Kerningan having to stand up every morning, providing updates on what bug or feature they are working on? Which one would throw its mouse to the Scrum Master head first, when he would complain about their project "velocity"?
I have come to the conclusion that is not about the
methodology but the people. I worked in super productive teams that were not following any "agile" methodology, or really any particular methodology.
The goal was to get shit done and make customers happy.
On the other end of the spectrum, I have worked with teams
that were pretending to embrace agile and scrum methodology
though could keep standup quick and sharp.
It was an excruciating ritual that would take my most
productive hours of the day: every day!
I always try to work with smart people. People who don't need much direction and know how to get things done.
The article talks about big tech but it’s interesting to think of big tech in a couple ways. One is “Big Tech” (capital B and T) the other would be “big tech”. The destination is that the first is mainly the FANGs of the world that easily come to mind.
The second is other large companies that deploy huge amounts of tech but have different business models. Banks are an example. Apparently JP Morgan Chase’s annual tech budget is $12 billion!!! That’s a lot of money.
Name me one company that doesn't profile a programmer using the points and velocities (etc, etc).The up-speak and dishonesty can be seen readily when they say "All these metrics and data on your board will not be shared or used to decide your compensation or salary-revision.". The thing is that Agile, Scrum, Safe etc makes management happy that they have data which can infer devs productivity and return to the corporate. Thats all.
I’ve worked now at Facebook, tiktok, and snap. None of my teams used tickets so there are no points to track. Most of the time my tasks were just discussed with my team and main way you could track is by prs. How many points a task is worth is something I’ve only had to do for 1 job at a startup where we did follow scrum. I do occasionally give estimates but they also aren’t really tracked as they’re frequently not written down anywhere and just something I tell my teammates.
Startups implement Scrum or nothing, but the FAANGs each have an existing unique process with a cargo-cult veneer of Scrum on top.
All Scrum really means at a big co is an extra morning meeting that penalizes both people who like to work late and parents who need to get their kids to school. Plus you may occasionally hear the word "sprint" but it doesn't mean anything as you can be given an urgent task or have a feature cancelled at any time.
The important distinction here is this is a set or principles for agile software development at scale. If you aren't "at scale" and having trouble running basic Scrums, then you aren't solving any problems by scraping Scrum necessarily. If you struggle organizationally to use Scrum principles then you're going to struggle even more with this.
Pardon me for blowing my own horn, but as an alternative to all the usual approaches (Scrum, Kanban, SAFe, LeSS, etc.), for software engineering management at scale, my I suggest my own "TameFlow Approach"
It is based on Patterns and the Theory of Constraints.
Are Skype and Whatsapp even comparable? Whatsapp was competing with SMS (at least in developing countries) and it had a killer feature of making groups. Anyone who had a phone moved to WhatsApp overnight as they didn't wanted to pay for SMS.
Being ISV I do not do SCRUM but I did see the examples of it at couple of my clients and did sit on few SCRUM meetings there (never again). My impression was that is is nothing but feel good tool for managers to cover their asses.
We adopted it a year ago and the team still quite likes it. Reduced day to day stress and distractions and a good structure and shared vocabulary. Wouldn’t say it’s perfect but much nicer than the sprint treadmill.
We just started using it and I am pretty happy that it keeps PM (me) really shaping things with a fat marker to farm for dissent early on without distracting devs in endless meetings. So sharing vision and customer problems early is great.
What miss is (maybe its me): How do you get into that execution mode from there? Break it down? High level estimates? How do get started (especially large endeavors that would be more than 6weeks)?
I'm armchair at best, but I think scrum is a management and organizational process, something to help communicate upwards instead of something that is best for a team. That said, some companies and teams need the structure. They compare with Whatsapp, which seems to me like a small, dedicated, driven team with ownership and complete control over their own application. In my experience with Scrum, it was a product owner and management layer operating outside of the team, sending work downwards to the scrum teams while the scrum teams communicated progress and the like upwards. Less ownership, more oversight, etc. The scrum teams were implementation teams, there to do what they are told to do instead of owning a product. Cogs in a wheel, whose individual components (and the whole cog) could be replaced, shifted around and moved without the layer above really needing to worry about it.
I have been a proponent of Agile for many years. But it bothers me that it has been used to take power way from Senior Developers. We really need to bring back the Tech Leads.
A lot of large companies in banking or insurance for example, don't understand or seem to realize they are going to have to execute as if they are tech companies.
Kind of surprising that the article doesn't mention the profits and sway that Big Tech has to afford the approaches he describes. Borrow money very cheaply, buy back stock, profit from higher stock price. As well, Big Tech has market power (not necessarily monopoly) to allow for a lot of flexibility in deadlines.
I'm old enough to remember the B.S/B.A. (Before Scrum/Before Agile) times.
Agile has mostly been a disaster IMO.. I've worked at big public companies and 2-3 startups during it's run. It works to come up with a "MVP for 1.0 and then it starts dragging everything down because it encourages tech debt.
My experience has always been the successful companies/teams were tongue in cheek telling everyone they were Agile when agile was popular while simultaneously trying to ignore Agile as much as they can because it was mostly ceremony and downsides once a product gets anywhere near mature. There was a lot of "we're agile with a lower case a" at places that could tell the Emperor's new clothes were suspicious.
Some of the old processes have been slowly creeping back in, and we're kind of at the point now where teams no longer need to feel like they have to advertise saying they're scrum/agile to appear competent. But I think you either need some older managers or something else to figure out what to do next and how to bring back the good parts of older process methodologies because it feels like theirs a vacuum.
The relationships between teams and PM have changed so much. Agile elevated PM a huge amount but as Agile moved on PM seemed to abdicate all responsibility to do their part in the whole agile process, and now you're left with a bunch of extra PM just getting in the way.
Agile without developers being included in estimates was one of the worst things.. non-technical PMs constantly shooting down estimates with "No, that is easy" is the worst. But Agile to me always seemed to be about management being able to absolve itself of all responsibility for not being able to figure out what to build or how to build it and to always be able to blame the engineers.
As we go forward teams need to look back at the past and what worked before Agile, not just make something new up out of whole cloth.
One agile team I worked on which probably was the strictest essentially pivoted the entire product to something else 3x because PM never had a clue what to build that would actually sell.
> But Agile to me always seemed to be about management being able to absolve itself of all responsibility for not being able to figure out what to build or how to build it and to always be able to blame the engineers.
This is Agile’s “Executive Armor Effect”, which turns developers into a sort of ablative armor that can take hits for executives.
I just think Big Tech not using scrum is a strange way to make an argument against scrum since there are sectors in tech that ship faster imo. So why not look at the fastest shipping segments and see what they use.
I would love to hear from Big Tech employees if they feel like they ship a lot. To give a baseline, 1k LOC per developer per week is a lot. That's what the highest performing teams can produce.
Compared to who? Two-person startups can be faster yes, but all the Big Tech companies are legit racecars compared to (say) a bank or a pharmaceutical companie.
Also, shipping fast and actually innovating are not the same.
Matches my experience well. Scrum works OKish in small teams depending on how mature their tech-lead and product owners are. It seems the default in consulting companies where tech consulting companies work with non technical customers. I've seen a few good examples and also some really bad examples of its application.
As soon as you scale scrum to organization wide, you get inter team communication challenges for which Scrum has no clear solutions (because its a team methodology and not company methodology). Once you have multiple teams that need to align their plannings to get stuff done, inter team dependencies become a bottleneck. Add a few external teams to the mix and you get a perfect storm of conflicting interests, miscommunication, politics, etc.
Having a lot of dependencies between scrum teams also means you get a lot of decision making lag. Scrum teams have two issues: they work in sprints with some fixed length and you can't introduce new work during a sprint. If either is not true, you are not doing Scrum. When you have a graph of interdependent scrum teams, the lag you build up to get anything done is basically proportional to the path length through the graph. The bigger the org chart, the slower things get and it adds up quickly. The longer the path, the more planning and implementation sprints are involved in each team along the path and the more chance there is for things to get lost in translation. Beyond 3 vertices, the whole thing starts looking like a very chaotic waterfall style process where it takes months/years to even agree to do a thing.
I speak from experience. The most dysfunctional thing I've seen on this front was Nokia flying in teams from three continents to have a bi-monthly sprint planning about 12 years ago. That worked just about as well as you can imagine (i.e. not at all). You needed to get your requirements scheduled 3-4 months in advance to stand a chance in hell of anything happening under 6 months. I'm talking critical stuff getting de-prioritized because one or more teams involved got other priorities imposed on them and thus had to say no. One of two things happens in such organization (usually both): 1) things get escalated to VP level management, a lot, and there is a lot of politics with lots of egos clashing. 2) teams eliminate dependencies to each other to speed things up (i.e. they reinvent wheels they could have reused just so they don't have to deal with each other). Both are bad for productivity and cost.
The way out is to bypass middle management, ignore the process (on both sides) and establish direct lines of communication with engineers in different teams. Once you have that, pragmatic decision making can actually happen. At this scale, middle management should just not get involved with day to day engineering decisions and instead enable the day to day interactions and smooth out any inter-organizational bottlenecks.
Most of the big companies mentioned in the article are not great at this either. E.g. the reason Google has so many chat tools is because they have teams working against each other rather than with each other.
I’m familiar with the way of working described in the Skype web section. One thing this article has made clear is how much responsibility is pushed down to engineering, and removed from product ownership.
The product owner is not even tasked with approving the work done! Nor having any real kind of spec. However in my experience they still have veto power and the power to change what the project is at their whim.
This is all done with the idea of “autonomous teams” held firmly in mind. Yeah you’re autonomous... you can decide for yourselves how you will do what the project owner has decided they want today.
But they are still the owner! And will be the face of success when your project ships.
Everything described in there is mirrored in SAFe fwiw, even though their survey said it was mostly used in large non-tech companies.
Individual teams can use whatever they want to manage themselves (Kanban, Lean, Scrum, etc…up to the team). Estimations planning and commitments for each quarter are done by developers exclusively, including coordinating across teams. Releases can happen at any point. A strong and continuously improving CI/CD pipeline is an expectation.
I really wish more people knew about SAFe. It’s constantly improving and refining the experience and from what I have seen, if there is a better way to do things SAFe will become it.
SAFe is beautiful in that it is the ultimate expression of Enterprise Agile Cargoculting: all pretenses of agility are lost, there are ten different levels of decision making, and all responsibility is diffused in a confused mess of worker bees running around under an all-powerful and all-seeing LPM
> Individual teams can use whatever they want to manage themselves (Kanban, Lean, Scrum, etc…up to the team)
Except that they can't, because they will need to align on PI dates based on sprint durations, publish their burndown statistics and various other measures, and there's no way the poor soul who is reponsible for collating that will go into several different tools
> Estimations planning and commitments for each quarter are done by developers exclusively, including coordinating across teams
No they're not, they're done by the product owners, the system architects, the release train engineers (shhh... don't call them project managers), the product managers etc
> Releases can happen at any point
In my experience, releases can only happen at the end of a program increment, or maybe at the end of each sprint because until then nobody has integrated everything together
> A strong and continuously improving CI/CD pipeline is an expectation.
Fair enough, until the "innovation sprint" becomes the "bugfix spring" and all those good resolutions fly out of the window
> I really wish more people knew about SAFe.
So do I, just like I wish more people knew about colorectal cancer
> It’s constantly improving and refining the experience and from what I have seen, if there is a better way to do things SAFe will become it.
SAFe is constantly evolving, because that's the easiest way to keep the money coming in. It's as if PADI had found a way to change diving equipment every 2 years and ask everyone to recertify
One thing I've noticed with Scrum and SAFe are that all negative experiences are not _true_ scrum/safe. It seems entirely natural that processes will shift as team composition changes or the rules chafe. Once there's been drift from the canonical process there doesn't seem any way out of that except doubling down, possibly by buying more training.
Judging by the number of SAFe consultants (complete with that video of a guy drawing comic-book style diagrams about spotify, cute the first time but infuriating after the 10th) I'd have to say that if our SAFe was bastardised maybe SAFe itself is a bastard
I know many companies running SAFe on the east coast and it’s generally an improvement across the board from everybody I talk to. The only problems I’ve seen are from 2 camps: the people who don’t want to have to plan anything and the people who want everything that is planned to be 100% accurate and not a best guess. Either of these two groups will cause problems to a process like this with the latter usually being in some type of leadership role.
SAFe consultants or not, it’s still up to your company leadership to implement it. A lot of SAFe consultants I have talked to try to advocate for things that follow Scrum because many teams are already familiar with it. This is honestly one of the biggest issues that I have found with the teaching.
80% of SAFe is based on Donald Reniertson’s “2nd Generation Lean Product Development” book, which is fantastic. Once you’ve read that it’s easy to spot the differences.
I hate that diagram. It looks like gibberish to anyone who hasn’t gone through a training to understand it, and even this it still looks like gibberish.
SAFe is about addressing the communication problems that happen as an organization grows. You’re looking at the diagram for everything that would be applied to a very large organization.
SAFe itself has a goal of using 50-150 person “release trains” that are essentially independent units. That’s the core of what’s going on.
Everything outside of that is about managing communication between other “trains”. In the diagram that you’re looking at, the bottom 2 sections where the CD Pipeline sits is where most developers will experience SAFe and you’ll find that it simply advocates for DevOps best practices.
I completely understand how everything else looks like a mess. It’s a lot to digest.
Also note up in the top left corner, where you've got the customer abstractly feeding work into this treadmill of... stuff. If your development team aren't talking to either the customer, or representatives of the customer, nothing else in agile will work.
SAFe is just waterfall management with agile keywords tacked on. It's not an engineering methodology. The only "problem" SAFe solves is reintroduction of deadlines and quarterly objectives, which flies in the face of everything agile actually stands for.
The quarterly PI’s are for planning because the most you can reasonably plan for with any degree of accuracy is about 8-12 weeks.
There are no deadlines imposed. You plan what you’re going to work on and how long it will take. You coordinate with other teams for any dependencies they may need from you and when. But you as a developer are responsible for saying how long it will take and when you can have it done.
Even after that, you’re only allowed to plan 2/3 of your time and the final 2 weeks of the PI are unplanned specifically to build in more buffer for you. If you do finish when you originally planned to, that 2 weeks is equivalent to 20% time where you can work on anything you want for the company, training for yourself, etc. It’s a built in reward for being on schedule.
And then as a group you discuss all of the risks that could prevent one of these items from being hit schedule wise. So if something comes up that keeps a team from being able to finish something, everybody was aware of it in the beginning and had the opportunity to try to help address it.
There’s no waterfall to it unless you consider anything that acknowledges dependencies to be waterfall.
Planning for 8-12 weeks at a time is the ideal window that lets you set expectations, coordinate multiple teams and avoid derailment without the idea that you can plan forever into the future. The farther you get from now, the more any plans become pure imagination which is why year long waterfall plans are a joke. It’s entirely possible to plan 8-12 weeks at a time. Especially when the point of the system is to acknowledge that estimates are at best a guess.
There’s a balance between setting expectations that business people can plan around and working in a box without communicating anything other than “I’ll let you know when it’s ready.” SAFe strikes that balance better than anything I’ve seen.
But everything will always boil down to leadership embracing it. Until leadership can acknowledge the fundamental lack of accuracy from software estimates, you’re screwed no matter what the methodology.
> You as a developer are responsible for saying how long it will take and when you can have it done.
Which amounts to estimating software deliverables beyond the scrum window, which will invariably lead to disappointment, which will eventually fallback upon the shoulders of devs. It is impossible for delivery dates beyond the estimation horizon to not start being used by management as an anchor at some point in the near future. It only takes one manager promising stuff for the rest of the management team to try and emulate the savior.
Teams coordination only requires timely backlog priority management and does not need to be planned top-down for days on end with all members of all teams present.
I went on a scrum course and the takeaway was basically that feedback is a big deal, and you should try to get some repeatedly and quickly. It's also common sense that you should have tasks written down somewhere, and that some of them are more important than others.
You certainly need to think about how long tasks will take, but there's no reason why you need to do planning poker, that just seems to be one among many ways to think about how long something might take. Tracking velocity is another one of these things that seems replaceable.
If you have a team of people that more or less adheres to a few principles, there's no reason you can't get things done.