Rapid release means I can no longer rely on the software that engages in it. It's constantly shifting, with frequent changes in UI and workflow, and new and exciting bugs constantly appearing.

I miss the days when I could plan ahead for new versions, so I could time them to have the least possible disruption to my life.

The classic "eBay color change" comes to mind:

https://kulor.medium.com/how-ebay-secretly-changed-their-bac...

However, not every code release needs to include a UX change. I agree that changing the UX every week is a solid anti-pattern.

While there is value in discrete and planned releases, we value responding to change more.

But there is no rule that says you have to do those experiments in production and annoy your paying customers who don't want to see them. If you didn't insist on releasing every five minutes you'd have time to iterate privately. Successful software was made this way for decades before modern web applications came along and frankly a lot of it had better UI/UX than a lot of the junk that gets pushed out today.

While there is value in discrete and planned releases, we value responding to change more.

I suppose the question is what change you're responding to. In this situation it seems like a lot of that change and the need to respond to it are self-inflicted damage.

Personal favourite was Spotify removing their playback widget to "improve the customer experience" and oh my God the backlash was ferocious. It was entirely moronic and screamed of Product Manager -> Customer with zero actual user feedback gathering:

https://community.spotify.com/t5/Android/Android-Widget-Remo...

26 pages of complaints produced this response, basically we removed a thing you love and use constantly, but you can vote for it as a feature request:

"We always take retiring features in Spotify very seriously, and do want your continued feedback. The best way for us to measure your comments are via votes in our Idea Exchange.

For this reason we're going to lock this thread. We encourage you to move to the Idea Exchange area where you can vote and comment on feature requests."

Approaches and terms like "iterating fast", "responding to change", I dunno, just feels like a poor substitute for better planning and evaluation because those can be hard.

People are so obsessed with scale and techniques like A/B testing today that they seem to forget that kind of test can only tell you whether one of the options is significantly better. It can't tell you whether you're finding the best choice between excellent options or the least of evils between dreadful ones.

Microsoft seems a company just oblivious to user feedback - Beta users howling about disastrous Windows changes mooted, and they just.... ignore them. I don't get it.

I feel like this is the key point that adherents of modern development practices need to address. We've been sold a lot of stuff on the basis that it would drastically improve the user experience - but a lot of people, myself included, feel that UX actually declined in the past two decades.

Unfortunately this seems to have created a culture where little that ships was really ready - at least not to the standards we used to consider acceptable. Everyone is busy pushing out new features and other visible changes every ten seconds. But somehow no-one ever finds the time to go back and fix the substandard quality that got an MVP in front of users on launch day.

Google even warned us about this years ago. Remember in its early days everything was Google XYZ Beta? It was an excuse to ship stuff with substandard quality then and it still is now. We're just less honest about it now.

You can’t have an intuitive UI if it’s constantly changing so designers look productive in sprints: you’re constantly defying the intuition for your current product, while giving users no stable target to learn the behavior of.

You can definitely have both intuitive and changing UI if you think about it.

This forum gets hacked into that, you can't see points a comment gets unless you measure it by the gray of the comment, in which case you can tell exactly when it has 0 points, or -1, or -2, or -3, or -4. Scrapable, and illegible. Opens itself to voting rings. Not even legible if highlighted, you need to copy-paste to another application.

I brought this up formally.

There’s something terribly wrong in the picture you’ve described. I do not know what Agile stands for, but our company was doing “a new version each Friday” for like 15 years. One of the axioms of such a fast turnout was “a new version should work in the old environment exactly the same as the old version” - except bug fixes of cause, some of which were made optional nevertheless.

Small incremental change is easier to test in most cases, the issue is, changing software that was designed as a whole over to incremental feature improvements is hard, because often the existing bulk of code cannot be meaningfully unit tested.

Today it's just the wild west of people basically having production not far sepeeate from development in terms of changes. UI/UX changes are the worst in this context. I opened Venmo the other day and I had to waste 5 minutes because someone decided they wanted to change the UI and I had auto updates applied (shame on me I suppose).

Do not change the interfaces with your users folks unless it's truly broken.

You may have more patience and composure but most of everyone is sifting through countless activities in a day. 99% ill close the tab than wait for a 5 minute update for an activity im not getting paid to do.

But that doesn't keep programmers employed, VC's making money, or management justifying themselves.

Frustrating.

All I needed them to do was improve performance for large libraries.

The irritation is tripled if it's an un-skippable "let us walk you through the nineteen changes in a slowly constructed slideshow!"

This is second only to applications that don't properly retain state--especially after an update--so every time I open them I get a brief glimpse of what once was (and what I probably wanted to go back to) before I get a 50/50 split of being dumped to the application home screen or "oops, something has gone wrong! lol, software, yannow! Please to be logging in once more, thx."

    >click next.

    >click next

    >click next

[support . apple . com]

"Hello, my inquiry is whether or not AppleCare+ covers spontaneously throwing my laptop out the window thus it falls down three stories into a storm drain. No, this is purely hypothetical. Yes, I'll wait while you research."

I'm strict on this; I return video games with unskippable tutorials. It's cause for an immediate Alt-F4 and return to store.

Many features are better off hard to find and left that way than shoved-in-your-face discoverable, or introduced by a tutorial. Those who want to get the most out of their tools will find them and will appreciate you treating them like fellow intelligent beings. The rest will never know what they missed and won’t worry about it either.

Lots of game developers have conducted extensive research and found that player retention is considerably higher when you "annoy" them with these things rather than just throw them to the wolves and tell them to RTFM.

There's basically two reasons to make games: to make money, or to make art. You can try to do both at once, of course; and focusing on one doesn't necessarily detract from the other.

Consider FromSoftware, for instance; often pilloried by the gaming press for their lack of difficulty scaling or tutorials, but adored by their fans and able to turn a tidy profit with each release. Similar, but less well-known, would be Spiderweb Software; which has steamrolled along for decades without breaking their artistic style.

Not everyone wants to be the next Candy Crush or Clash of Clans; many just want to create an experience that they would enjoy.

From Software games are more hands-off because their goal is to make the player discover the mechanics and systems on their own. Not every game wants to do that, and as such I don't think games like Halo: Combat Evolved or The Legend of Zelda: Ocarina of Time are automatically worse for this. In the latter case, I think the unskippable tutorial section pulls double duty in setting up Link's backstory; You're with him as he first learns how to fight, use his shield, etc. before going off on a grand adventure.

From Soft games are also not immune to the unskippable tutorial mission. Both Elden Ring and Sekiro start you off in a relatively safe linear level with pop ups teaching you their basic mechanics.

I want to circle around back to the comment that started this though, because that user said they immediately uninstall and refund any game with an unskippable tutorial. Maybe they meant this as hyperbole and I'm just sitting here wooshed, but it seems like an extreme stance to take.

Indeed, tutorials do come in a great many varieties, and when it's done well the user may not even be aware that what they're experiencing is a tutorial. That has a strong tradition in game development; where a new ability or a new tool is provided to the user and the level is designed thereafter to allow the user to explore that new thing. That's closer to your Zelda or Halo.

Even FromSoftware games take this approach. Elden Ring has a whole opening stage that's clearly designed to expose users to core gameplay mechanics.

But then there's those unskippable dialogues, or the tedious "now press this input to perform this action" prompt found within a purpose-built tutorial area. Those, specifically those, are what I find cause for immediate product return.

There was some Need For Speed games on the PSP that were particularly awful for this. You'd be _racing_ and it would pause the gameplay to show you a dialog about how to drift or draft or somesuch; completely breaking the flow of the experience. That's what I would demand a refund for.

That said, art is subjective, and the experience is ultimately defined by the beholder. Maybe some people actually enjoy being brow beaten with unskippable prompts and hand-held through tediously constrained environments, with their experience jarringly interrupted at unexpected times by dialogs that break the suspension of disbelief.

The Northern Asylum in Dark Souls is one long unskippable tutorial.

The tutorial is basically the intro video with some tool tips periodically in game.

Which is why, for the most part, I've stopped using web-based applications.

Edit: not sure why this was downvoted. Did someone find the idea of a narrowly-focused phone to be offensive or unworthy of discussion?

We're not sure why you went to describe how you read emails, and that's why we downvote you. Your answer was completely irrelevant !

Interesting, I don't have a google account associated with my phone. It means the google app store doesn't work, but everything else works fine AFAIK. I use FDroid or just download apks directly.

Rapid release and experimentation means ultimately finding better solutions for the users faster. If you go slow, things don't change as much but they don't get better much either.

We tend to assume that all of the good changes would have happened anyways and all the bad changes are because of rapid release.... but that's not how it works.

Better UI is not universal. And the absolute, hands down unquestionably best, UI that can possibly exist for any given customer is the one they're already an expert in.

Can a different UI end up being better for them? Sure. Will it be better the day it's released? No.

The user has to pay a cost every time the UI changes, regardless of whether it's actually better for them or not (and it will never be better until much later, once they've become an expert in the new version).

Worse - I see more and more companies optimize UIs for the "initial user experience" and onboarding new users (more paying customers). The sad truth is that many times that means dumbing down a much better UI for the "already expert" customer - just like putting training wheels on the bike is great for the person that's never biked, and utterly infuriating if you already know how to ride.

This is why existing customer bases are so rarely fond of any change - regardless of whether they end up liking it in 3 months time. It has a real cost, right this second, and they're being asked to pay it.

If you already spent months training a new hire on how to get something working with all the different random tools and software they use, changing something meaningful means breaking that persons ability to do what they need to do for their job, at least for some time.

Even changing something inconsequential will confuse some users for a bit, require making/updating new training guides, etc.

If it’s a business on the edge or under during a stressful period? That can literally tip it over and make it non functional. That is especially true for small businesses.

This is one of the reasons business productivity software becomes unwieldy after a while -- it's fine to add buttons and menus, you just can't change existing ones or what they do. So then version 5 will finally introduce the massive, desperately needed UI refactor after ten years, while plenty of clients just stay on version 4. (An example of this is the Blackbird CMS -- old Blackbird vs new Blackbird.)

If your new product with ten years of improvements was so unattractive compared with the existing version your customers are currently using that they were choosing not to update, what does that say about the value to the customer of all those improvements?

In the world before SAAS we saw both extremes often. Some products had incredible dev teams and people would upgrade relatively often even when each new version cost extra money. Other products barely produced an interesting change in several new versions over several years and users mostly didn't pay for those updates.

SAAS is a convenient business model for the developers in the latter category because it locks in the ongoing revenues whether any future changes add value or not. They just need enough lock-in effects that once you've signed up you can't easily move to one of their competitors.

Then, switch to new vendor or the next major version.

Software Lifecycle for reals baby!

It's also built with table elements. The programmer in me is screaming because that's not the proper use of tables (I know that was the way to build layouts in the past, but that was then and this is now). The UXer in me is screaming because it's a nightmare for accessibility.

I want to add that I'm not coming from the "Overengineer-and-Bloat-Every-Website" UX School of Thought, so I agree with your point about Reddit. Being performant and light on resources is part of what makes a good UX, but that's the thing: there's more to UX that loading fast and running without hiccups. I think HN's experience could be greatly improved without compromising in any of those aspects.

The default font size is unreadably small, it's impossible to reliably tap on upvote without accidentally hitting downvote on mobile screens, comments with code blocks with too many characters in a line break the width of the interface, huge comment threads indented 15 times aren't collapsed by default, downvoted comments lighten to become unreadable, I could go on.

But there's no decent alternative for HN's content and community, so we put up with the terrible decade-old interface because we don't have a choice. Although people are constantly inventing their own clients for it precisely because it's so terrible.

The only of those complaints I even half recognise is the voting buttons on mobile, but I miss them rarely enough that I’d probably still rather not give up any more screen real estate (horizontally or vertically) or add an extra click.

Otherwise HN has basically the UX I wish every site had.

Frankly, neither programmers nor graphic designers are qualified to make good UI decisions. Human factors is a real field that people get PhDs in. You really can't just wing it, especially if you're a programmer whose mind has been thoroughly addled by decades of using programmer-designed UIs from the Nth circle of hell.

If HN had such great UX, there wouldn't be multiple browser extensions (and clones) that focus solely on fixing the UX. I'm not saying the UX is terrible, but I wouldn't call it great. I think the best term for it would be "consistent." It's basically Craigslist. It's fine if you've been using it for years and have adjusted your behavior to work with its quirks, but it does not provide a smooth experience that would work for a community you're starting today.

Just because there are other products with different experiences that skin it doesn’t mean the core product is bad!

It actually means the opposite - the core product has such value and there are so many users, it actually is able to support sizable enough communities of people ‘outside the core’ to justify niche repackagings for them!

Great UX should also mean fast, efficient, and dense with information. Respectful of time, attention, and intelligence. Not making an assumption of stupidity on behalf of your users.

Great UX should primarily mean those things, at least for a certain subset of software.

Making links in profiles and text posts clickable. That's like the most basic thing you can do for your users.

Adding the ability to collapse a massive comment thread without having to scroll to the top. Also, indent guides (especially needed for the massive threads around here). Shit, HN didn't even have collapsible comments for the longest time.

Being able to comment inline and not having to navigate to a separate page where you lose context of the surrounding discussion.

Highlighting new comments since you last viewed a thread.

These are all basic things that HES addresses, there are many more I can point out. I'm not trying to insult the admins, I think they do an amazing job of running the site, but let's not pretend HN is some pinnacle of UX design. We are all just very used to it.

And personally, in the years I’ve been here I’ve never noticed or cared about any of those issues. shrug

Adding a ton of features almost nobody uses and rearchitecting the site using 'modern' frameworks, only to make it 10x slower goes right againts that oath.

The funny thing is that HN doesn't even adhere to the basic UX practices that many around here constantly complain about in new design like knowing what is clickable.

Take a look at each story list item on the front page, what do you think is clickable? Everything in the subheading is formatted exactly the same, but not everything is clickable. Same goes for the headings of each comment in a thread. And I think HN's violation of this is especially egregious since the formatting of these things aren't just the same, but they are greyed out. That's a pretty universal sign that something is not clickable. This is 100% confusing to new users.

This one is a bit more difficult to implement without a lot of JS, but one thing that I have run into many times is reading through a comment thread, trying to reply, realizing I'm not logged in, logging in, losing context of where I was, scrolling through the thread trying to find where that comment was, giving up and not commenting.

In addition to that, not remembering which comment threads have been collapsed. Accidentally navigate away? Good look finding your place again, especially if there were many new comments added.

The mobile experience can be summed up as "very precise clicking"

I think the top menu items are also named in a weird manner. Why does comments take you to a list of random comments in chronological order? How is that useful? You would expect that link to take you to your profiles comments, but no, that's the "threads" link. There is a lot of that on HN where things are (imo, purposely) obscure. I get why they would do it, but let's not pretend it's a good experience.

HN caters to the most technically literate community on the internet. They will figure out how to use this site no matter if you had to navigate the site via the terminal (some would probably prefer that "NO JS!"). We've all gotten used to how the site works, we ignore all of its issues because we've adjusted our habits to work around those issues.

And I'm not saying HN has terrible UX (there are actually many things we can learn from it) but it's definitely not the site I reference when I need inspiration for a good experience.

I would disagree that UI constantly improves, though. I can think of lots of examples where the opposite is true. But that's not due to rapid release, that's due to other issues.

> If you go slow, things don't change as much but they don't get better much either.

It's hard to think of when I've seen improvements that were so good as to be worth the cost, though. Things certainly change, but change is not always better, and even when it is, it's not always worth the cost to the user.

Decades of “experimentation” have not improved the act of typing out a document in any meaningful way. Now, they have improved the experience for the corporations who can profit selling out their users data to the highest bidder and locking them out of their own devices while charging a subscription fee.

Chugchugchugchugchugchugchug

Watch icon

Chuchchugchugchugchug

Disk eject sound

Please Insert the disk ‘Docs 1’

Clunk

Watch icon

Chugchugchugchugchug

Word count: 2351

More responsive? No. No it wasn’t. Disks were slow. Memory capacity was low. You used to have to use third party RAM disk software to complete some tasks in a reasonable time. Or install a ram doubler to let you work on reasonable sized documents at the expense of slowing your machine down even more.

More intuitive? Hard disagree. The example I gave above was of getting a word count. You had to find the menu option to do that - it wasn’t just visible on screen. Same with spellcheck. Global search and replace was slow and clunky and felt risky. You only had one level of undo.

Faster to startup? Hells no. Again, slow disks.

More reliable? There was no auto save. Everyone who used a computer in the 90s has a story of losing hours, or even weeks of work to a crash. Everyone.

Easier to learn? Learn how? Word came with a few hundred pages of dead tree manual, but if you were on an institutional machine or running a ‘borrowed’ copy you didn’t have that. There was no internet to look up how to do stuff. Maybe you could pick stuff up from the tips section of a magazine? Sure. It was super easy to learn.

Compared to all of which, google docs or Word365 is just… light years ahead.

Most changes to already functional software are just changes. They aren't improvements or regressions; they're just making things different.

But yes, if you are regularly changing your UI in order to improve your system and roll out changes, you need to validate the impact. When I was last a PM, I spent a bunch of time monitoring the rollout of changes via Pendo to understand BOTH, if was getting the adoption of features I expected from certain cohorts and seeing if there where any shifts in user behavior (either organically, due to intentional or unintentional changes) And we did this for "beta" groups as well to really validate that feature were working with early adopters before broader rollout. I guess you might not consider the later "production" in sense, but these were real customers with real workloads.

You'll be able to find examples such as that, sure. But that doesn't make it generally true.

CounterCounterPoint: If the software was perfectly functional two years ago, all that can happen with a change, any change, is to make it worse :-)

In my experience it feels about 50/50 for better/worse, whether that is features taken away due to some business concern, blatant contempt for existing muscle-memory, or misguided designers reworking things that did not need a rework.

Given the tone of the message, I expect "non rapid" to mean with a much longer time frame, and, most importantly, the release must be controllable by the user.

This is pining for a world which no longer exists, because security patches must be applied to anything that matters immediately.

Of course you do need a strategy for maintaining multiple versions of something in that case instead of just forcing the latest version of everything you do on users whether they want it or not. With the lock-in effects of the always-connected web applications of today we all know which option makes more money.

I take some comfort in knowing that eventually most of the developers that force unwanted changes on their users will probably see their market share erode and lose their dominant positions. It's just unfortunate that we have so many quasi-monopolies in both essential and niche software products that it's going to take a long time for that to happen.

This world does in fact still exist, this is how most OSS works!

CD is a deploy strategy saying the aspiration is to do one deploy per commit, rather than batching commits and deploying once per N weeks or whatever. It has approximately zero to do with UX and product design/iteration approaches.

If you're not familiar with the general arguments for why CD might be desirable, a good place to start would be the books Continuous Delivery [https://smile.amazon.com/Continuous-Delivery-Deployment-Auto...] or Accelerate [https://smile.amazon.com/Accelerate-Software-Performing-Tech...]. The TLDR would be that most deploy problems in practice turn out to be because the process is manual, infrequently-used, and/or unreliable, and by practicing "if it hurts, do it more often" you are forced to iron out those kinks. For example you can't deploy automatically without a good acceptance/smoke test suite, and without solid DB migration testing/linting/practices, to name just two of many. You could just build all these things without trying to do CD, but in practice it seems that it's quite hard for organizations to prioritize this work in isolation, and the goal of CD seems to provide a fairly clear map to a much higher-quality deploy process. The practice of deploying more frequently to flush out issues, is also quite unintuitive as you note.

Here's a (probably out-of-date) article on how FB did deploys in 2017: https://www.infoq.com/news/2017/09/facebook-release-scale/ -- batched commits and staged canary rollouts automatically cut and released every few hours.

If you have external contracts that are onerous to validate then it can be quite expensive to build high-fidelity internal representations of those contracts. Could still be ROI positive to do so!

In most cases the contracts are either internal or simple (non-temporal) external contracts like a REST API, so most orgs benefit from moving in this direction.

Doing this in production is being disrespectful of your users. Users are not waiting for your PRs to be approved and merged, they can be as delighted by your new button’s shade of lightgrey if it is released some days (even weeks !) after.

The connection is that if the commit changes how the product works in a way noticeable to the user, then the user must suffer the changes frequently. Since changes incur a cost to the user, it's requiring the user to constantly be paying those costs.

If the deployment happened less frequently, then the user would incur the cost less frequently, and perhaps in a way they could schedule when to take the change on so it is the least disruptive to them.

Most software nobody cares if it works or even is possible for untrained humans to use, and it shows.

1. Release often

2. When a release goes badly (and they will) do some version of '5 whys' and fix the root cause.

3. Continue to release often

That step 2 is often missing and what usually happens is that no engineering time is committed to the solution. Instead they implement more process which costs nothing, doesn't solve the issue and slows development.

Just this summer, we went through these bugs one after the other:

1: Putting the ATV to sleep after finishing a video results in in a black screen with the video's audio playing from the beginning. This can be rectified by backing out of the video.

2: Putting the ATV to sleep while watching a video will cause the video output to freeze for 10 seconds or so when the ATV is turned back on. Eventually tvOS recognizes the app as unresponsive and kills it.

3: "Right" on the circle pad can no longer open the subscriptions tab, even though it still works on all other tabs. You have to use the center button.

4: When starting video playback, the first several seconds of sound are cut off, requiring the user to rewind if they contained anything important.

I don't think any of these issues existed simultaneously. Issue 4 started happening with the exact same release that fixed issue 3.

New releases should almost always only add features or fix bugs.

though looking at the history the number of recovery (point) releases have increased a lot

https://github.com/microsoft/vscode/releases

That's one of the many advantages of desktop apps: unless it's a bad player that forces the update down your throat, it is you who control if and when to upgrade, not the vendor, who may or may not have their interests aligned with yours.

(It's a bit similar on the mobile if you disable automatics updates, provided that you can ignore that obnoxious red dot forever.)

No one is going to say No to something new and sexy, especially not the customers (since we won’t ask them).

The trick would be giving users more distinct choices. For example, with something like Chrome the choice is between extremely frequent and just plain frequent. They don't really offer any long-term version.

This used to be a endemic issue for the mobile apps. And now even the desktop apps have started doing it. I am looking at you web browsers.

I’ve been working alone for 12 years and all those concepts were already working in the last large corp I was in.

What trends am I missing by being disconnected from the employment market? Am I missing anything or do we still develop like it’s 2010? Is there no new process or evolution on the style of management of software? Is Agile the final form of product management?

But they don’t seem as earth-shattering as missing the Agile boat.

From the article:

If I were to generalize, I’d say most of the companies I’m picking on fall into one of two categories:

    They believe in DevOps ideas, but they’re unwilling to invest in changing their ways.
    They aren’t convinced that some of the scarier ideas (like deploying multiple times per day) will help after all.

I’ve been thinking that UI changes should follow something like semver like APIs. UI is a contract between you and your users and completely shifting layouts should be seen as a breaking change.

These breaking changes should be avoided more critical the product is. E.g. health management app for elders.

And so there is no “staying on windows 7” for SaaS.

What would address my concern is if it were possible to skip releases, however rare or common they are, as I wish. But that raises an entirely different Bad Thing in the modern software industry -- that security updates are increasingly inseparable from other updates.

It started out well -- we spent a few months hacking with Terraform, Docker, Vagrant, Kubernetes, and related technologies to implement an infrastructure-as-code approach -- automating the process of provisioning and decommissioning servers, and building a setup where development teams could deploy updates with a simple git push.

Unfortunately it all went downhill fairly rapidly. We ended up spending the majority of our time manually applying security patches to a bunch of snowflake servers that we'd lifted-and-shifted from another hosting provider to AWS, and fielding support requests from the development teams. Within a year, we were being told in no uncertain terms by our project manager that we were an operations team, not a development team.

It felt like a complete bait-and-switch. Within two years, I had left the organisation in question and moved on to a new job elsewhere doing actual development again. Last I heard, the entire team had been disbanded.

It sounds like the author of this article must have had a very similar experience. I wonder just how common it is. It seems that in many places, "DevOps" is all Ops and no Dev.

This was definitely my experience at my last couple of jobs. At my last one they "scaled out their DevOps team" by hiring tons of juniors with next to no software development background. And then they "empowered" teams by assigning the juniors to each dev group. As a result, we ended up having to train them how to do their core jobs, which... went about as well as you'd think.

Eventually, there was an attempt to shift everyone to kubernetes. They had a special "DevOps" team build a layer on top of it to handle the non-kubernetes aspects of deployment as well, and somehow manage them together using Helm. If you're wondering "what the hell does that mean", well, it turned out nobody really knew. These "DevOps" engineers didn't really seem to understand kubernetes core concepts, and just ended up hacking away with some scripts on top of terraform delivered via Helm until something got configured. It was incredibly slow to deliver, hard to use, and I just stayed away from it until some exec threw down the mandates. (And then everyone started quitting because it was an absolute disaster.)

Ultimately, these are really stories about bad management, not really anything to do with DevOps. But that's how these things roll - some new hot concept comes to town, and bad managers try to adopt the term, without really understanding it.

When doing the market research, I talked to ~150-200 companies (mostly SMBs). Everyone was trying to "do DevOps". But the complexity of running a Kubernetes cluster (or a custom AWS setup using ECS) is just overwhelming for most of the teams.

In most cases, the DevOps/platform team requires atleast 2-3 experienced people that have successfully done this before.

Considering how few experienced DevOps people with such kind of experience are currently available on the market, it's no surprise that only the "coolest" companies around get to hire these people. These successful companies then write blogposts about how successful they were.

And the circle starts all over again. Less successful companies follow them and (in most cases) fail.

Most of these companies don't admit it, or don't admit it soon enough. They also don't write blogposts about their failures.

From my experience and research, roughly 70-80% of companies fail to deliver the expected results (or deliver them with order of magnitue more effort than initially expected). Yet 90-95% of the content we get to read about these topics is overwhelmingly positive.

PS.: If you don't have an A-tier DevOps teams, check out https://stacktape.com. I promise it will make your life easier.

And from the outside, this all just looks like resume padding and job-security. Devops is the new priesthood, subscribe or be reduced to irrelevance by config warriors.

Gets pushed onto them top-down. This means there is no real competition, no real empiricism, no real comparative merit involved in the switch.

How should never be imposed top-down. It's only goals and how success is measured which should be top-down. The classic example of this was when Jeff Bezos mandated that all Amazon software systems should be accessible by APIs over the internal network, or else one would be fired.

Whenever I've seen How imposed top down, I've only seen the lower level managers talk about how they could put off and passive aggressively stymie the initiative.

This has been my experience at 3 different small-medium companies now. A too small DevOps team suddenly is in the critical path for even the most trivial software task, then engineering productivity grinds to a halt. I think a much better pattern would be to enable dev teams to self serve. Set up the required infrastructure and guard rails, then let teams handle their own deployments and infrastructure. Give people what they need to do it themselves instead of having to open a support ticket for everything.

I think that's how DevOps is actually supposed to be done in the first place. You view Ops -- and the code used to manage and support it -- as a product, and get a specialised team of experienced Devs (and architects) to build it.

Once you've got the basic infrastructure and architecture in place, you then train up the individual development teams to customise it, extend it and troubleshoot it as they need to. In much the same way as they do with any other software product.

Most organizations don't have the resources, mindset, or skills to support a software library product and should only do it as a last resort.

If the dev ops team is staffed enough to develop integrations that won’t allow that AND won’t get in the way, and then train folks and ‘keep the line’ enough to stop the scope creep - they’re probably not at a shitty small/medium sized shop.

  1. An operations team with a different name.
  2. A platform team with a different name.
  3. A development team with a different name.
  4. A "CI/CD team".
  5. A role (ex. "dev who automates ops", "ops who codes", "support specialist who codes", all three in one).
  6. A chart that the delivery manager maintains.

  1. Delivering rapidly and consistently with extremely high levels of confidence.
  2. The right people address problems correctly, immediately, the first time, and fix it so it doesn't happen again.
  3. That's it.

Increased velocity is what business get promised, yes. It’s what you want.

The reality is that you can’t just magically make that happen.

Draw a line. Now draw the rest of the owl! Easy~

The problem has never been understanding what the desired state is… it’s always been that getting from the current state to the desired state is very very hard, and continually road blocked by:

- people who don’t want to learn new skills

- people who don’t want to seed control they currently have (over process and product)

- a lack of clarity on who is responsible for what systems

Devops is a load of hype.

There’s never been any reliable process to move to rapid delivery from it.

Yes, some teams have managed to get something that works, and there are a lot of tools and a lot of training which has resulted in over all better SRE processes.

…but by and large, that’s because of using better tools (eg. infrastructure as code) not because of devops.

When was the last time you had a “devops” guy you got to do something for you?

Right. That’s ops.

When was the last time something broke and the people responsible for making sure it never happened again were the “platform team” or an SRE?

Ops again.

You had all that before you started your devops journey.

It’s all just ops, with slightly better tooling, less outages, higher reliability and absolutely zero increase in product velocity.

Devops was the promise that by bridging operations and development you could get high reliability and faster iteration by having teams that could “cut through” the red tape and get things done.

That appears, largely, not to work.

Yes, developers that understand systems tend to build more reliable software.

No, it is not faster to do it that way, and the transition will be painful, and, because businesses mostly care about iteration speed more than reliability, even a technical success, it often a fails to deliver on its business value.

I've watched this particular breed of organizational cancer destroy many companies and products.

The end game is that people creating useless, but highly visible churn get promoted, as do the ones that repeatedly break stuff. Even if that doesn't happen, the engineers that want to build stuff inevitability flee.

It's also worth acknowledging when you don't need DevOps. Banks, for example, shouldn't need it. Their entire purpose is to be slow and reliable. Most of their money is literally just old people keeping lots of money in one place and not touching it. They shouldn't need to churn on features and ship constantly.

Something is either old and nobody feels like fixing it or something doesn't fit into the current constraints of the platform featureset. So they build something on their own, probably undocumented and without consulting the supposed DevOps team, but still using parts of the platform that have not been declared an API. But when you want to exercise the freedom you supposedly built with your platform, all these edge cases fall back on you and inhibit change. "We can't change the ingress controller, we rely on this implicit behavior of it" "You can't change how database credentials are provisioned, we pulled these from the Cloud SQL console and comitted them to git". And facilitating any change as soon as you can cover the use-case is a fight with stakeholders and POs that I usually have no nerve for. "Why do we need to do anything??? It works?". And then you get blamed when it breaks. I love this job.

Here is where it went wrong. The app should be re-deployed in a proper way, full infra as a code. No patches, new AMI instead.

While DevOps does work in some organizations, sometimes what you really need to do is operationalize your software development team. Don't get me wrong, system developers/engineers/operations are needed but for sometimes for a small four person start up, the best investment is improving the development team's operational posture (e.g. run books, alarming, monitoring).

Oops, I thought that was DevOps.

This is also where you will find things like a "CI/CD" team that doesn't do continuous delivery: they just build Jenkins pipelines, and run them as part of the manual cluster bringup every 2-4 weeks. Or "RESTful" to mean "APIs which use JSON and HTTP verbs" regardless of any hypermedia. And let's not even get started on "agile"...

Maybe we need to come up with a new term to be more specific. “DevOps for developers” vs. “DevOps for operators” or something like that. (DevDevOps va OpsDevOps?)

I always took DevOps to be some combination of 1) applying dev best practices to ops, which could still in theory be done by a separate team, but more 2) unifying dev and ops in one team/role, perhaps enabling developers to own it by providing tools and abstractions that developers can work with, and workflows that unify the processes and improve collaboration by lowering cycle time.

It seems that the “bad” DevOps is OpsDevOps where the old ops team just gets some rebranding, and maybe some new tools like Terraform. I’ve always found the “DevDevOps” approach to be quite empowering. Giving developers a tool like k8s so they can abstract away the hard half of ops, and understand the rest.

In my experience it went poorly because us devs are now handling our own deployments and building alerting systems and such, but the "devops" people won't give us the level of access necessary to do that with tools designed for the job, so we end up building custom ops-related tooling into the app and writing backdoors that let us use it without having wake the ops folk.

When you're accustomed to having to build everything you need, and you're tasked with ops work, you end up reinventing a lot of wheels in a rather weird way and compromising your app in the process. Or at least that's what we did.

Even in information security it basically revolves around the same things before you can even do any sensible work and improvements to processes.

  - Know what you have
  - Know where it is
  - Know what it is doing
  - Know what it is supposed to be doing

That's because marketing why you should know the what/where/how/why of your gold to business decision makers is apparently much more palatable than abstract concepts found in IT, even if the value is the same.

If you get the resources (including time, money and manpower) do just do it, it can be done. But you usually don't get all three (and maybe not even one), and therefore you're stuck with a task that cannot be executed well.

Fatality rates dropped, but it basically froze the technology and capability into the late 70’s/early 80’s and locked in just a few companies. The overall number of people working in that industry is tiny now. It’s almost a complete dead end.

It’s the classic trade off - cheaper, better, faster. Pick any 2 if you’re lucky. Most only get to pick 1.

On one hand we don't all want to buy mainframes, on the other hand, doing it as cheap and unmeasured as possible makes everything flakey, demanding on individuals to keep it running, insecure, and opaque to everyone involved. Too bad there isn't really any middle ground, it's almost like it's either "pretend to be a mainframe" or "do it as cheap as possible".

Regarding the aviation analogy: it definitely doesn't become cheaper due to regulation (it did become cheaper due to subsidies), but I'd rather have a system that's regulated (or forced) into safety than having it be optional and left to a sociopath (i.e. a for-profit publicly traded company - all the benefits of pretending to be a person but none of the responsibilities or limits to govern it - except the law and regulation).

  - Know what you have

  - Know where it is

  - Know what it is doing

  - Know what it is supposed to be doing

this is devops

Now DevOps means "We've created a DevOps team! They do DevOps!" Frequently its the managers of old school operation teams changing their names to DevOps teams to get a promotion.

"Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it." - Werner Vogels, Amazon CTO (2006)

DevOps meant that Developers are also responsible for Operations. But that meaning is lost to time, just like hacking meaning programming.

Isn't DevOps in large part forcing the developers to have responsibility over the code they push and giving short and frequent pushes to easily rectify problems?

Making developers do support work reduces support in the long run because they'll get frustrated and automate solutions.

My understanding of root of the term is about ownership and responsibility, where the team that develops the app also owns how the app deploys and runs in production, including monitoring and responding to issues.

If the team decides that the right way to deploy their app is once every two weeks using a shell script and a zip file, cool. As long as it's documented and monitored so you know if it's working.

Many companies, they don’t.

But that's literally what DevOps is? DevOps isn't a special DevOps team that writes terraform.

One thing that has changed drastically over the last 15 years is the sharp divide between IT and development. It used to be that developers were clueless how to deploy their own apps, and IT was terrified of writing code preferring to do things manually, or purchase something prepackaged. Now I see developers that understand firewalls and load balancers, and IT folk that can script stuff out for themselves. I think we have the push behind DevOps to thank for that progress, and while there is still a long ways to go, I think it's a good thing overall.

Something like is the kind of job that i think i want to do next. It need not be for a network team...but i'm headed more for back-end work and these days i rather engage with internal teams.

Separately, to your comment on sharp divide...The sharp divide that i have seen is a lack of cohesion for the following 4 IT groups: first-level support which gets sent to off-shore; the folks who (honestly have the most face time with internal clients who) roam around the office fixing desktops and building network/wiring, and setting up/fixing A/V equipment in conference rooms; sys admins; and developers. Furthermore, in the last 2 companies that i worked for, there has been a push to "centralize" certain functions like IT and HR to "shared services"...but at the end, none of these groups seem centralized! They all now work more separated than ever before...and most of the interaction between each of these groups - and even between the 4 subgroups i noted above in IT alone - mostly communicate via tickets, or some other thick-layered mechanism. Sure, they've been centralized in that these people were pulled from some respective business unit, and brought under a more single umbrella...but in the end, these workers hate it, the effecicny drops, more bureacracy is created (which creates no value)...and some executive gets to lead the shared services...and of course some management consultant company gets their payday. Oh, and yeah, IT at these orgs definitely does not use any of the good parts of devops, agilescrum whatever. Some orgs try and adopt some cookiecutter approach only to have it create yet more bureacracy...but "hey, we're agile!" /sigh

Every team had dedicated engineers whose sole purpose was to place purchase orders for servers, set up and manage databases, deploy changes, monitor uptime and more. And this was standard practice across the industry, as soon as your company outgrew a shared host.

Even the thought of something like this would be absurd today. So I'd flip the "no one admits they don't to it" around – your company does DevOps already, whether you realize/admit it or not.

Facebook calls it production engineer, DevOps is broad/ im unsure its origin, Google calls it SRE.

> the outcome of applying the most trusted principles from the domain of physical manufacturing and leadership to the IT value stream. DevOps relies on bodies of knowledge from Lean, Theory of Constraints, the Toyota Production System, resilience engineering, learning organizations, safety culture, human factors, and many others.

Anybody that's worked in an organization that "uses DevOps" or has "DevOps Engineers" knows that this simply isn't true in the vast majority of circumstances. Too often DevOps is just a new name for an ops team. "True" DevOps is all about aligning incentives, sharing responsibilities, and fostering collaboration between people who build and maintain systems whereas "DevOps" (with air quotes) is just a new spin on being a sysadmin with a greater emphasis on AWS and Terraform.

Historically, the word "DevOps" meant that instead of having separate "Dev" and "Ops" teams, the same people would do both Dev and Ops work.

Because they do both jobs, folks understand both sides, and work together to make things work better end to end for them and for customers.

Alternatives include siloed dev teams that throw shitty software that crashes in production all the time to an overworked ops team that takes all the blame.

And overworked ops teams that have gone pager-blind and only care if a VP yells at them, so ignore all but the most major issues - after they already happened.

No, it's not. The definition they provided for DevOps was markedly different:

> DevOps is [...] the outcome of applying the most trusted principles from the domain of physical manufacturing and leadership to the IT value stream. DevOps relies on bodies of knowledge from Lean, Theory of Constraints, the Toyota Production System, resilience engineering, learning organizations, safety culture, human factors, and many others.

If somebody reads that sentence - without already having knowledge about what "DevOps" means - are they going to think "oh yes, I see, so DevOps means having the same team do both Dev and Ops work"? Of course not! They're not going to have any idea what DevOps means, other than the fact that it's something consultants sell with word soup.

Knowing what the buzzwords mean, I was translating. ‘stewardess, I speak Jive’ style.

Ops is hard, and irritating, and a pain in the ass.

It’s also the process of actually solving the customers needs.

If someone can fiddle around cutting and pasting off stack overflow without having to worry about it, most will.

I personally can’t NOT be involved, but that’s why my resume looks the way it does.

To properly build and run a high-performing organization, you need to understand at least 1) the entire history (up to today) of software development practice and production operations, 2) all of the competing requirements of product design, development, and operation, 3) architectural best practices, 4) the industry standards that apply to the work, and 5) tech product-aligned business management methods.

Basically you need a generalist who has 15+ years experience and has tried and failed at everything under the sun. I think of it like carpentry before there was a building code, and before the introduction of apprentices and licenses. Trying to build high-performance orgs without a master builder to show you is like trying to build skyscrapers after having built your own home (and having never seen a skyscraper). Or worse, taking an old barn (non-tech enterprise) and trying to build a skyscraper out of that. Without a trade school it will continue to be just a craft.

Most companies that have sufficient talent and organization/operational setup don't see a need for DevOps.

There seem to be a lot of other companies that are sucked in by its promises and think that it will solve their issues. Surprise... this didn't fix your shitty policies, disorganized/unempowered org structure, or cheap talent (cause you're unwilling to pay or train).

I assume there are a few places that can benefit from this. Maybe some medium to large tech companies with great procedures, documentation, and an empowered org structure. Especially if they're paying enough for people who can manage it. The other is probably for very small places wirh fairly simple tech where you can't afford entire teams of people so you empower someone to really own the system top to bottom (without ever calling it DevOps).

You need some sort of Devops (best practices if not a team) to provide guidance and help implement standards for these.

When I was talking about small places, I mean like tiny mom and pop non-tech companies with an IT department of 1-5 people.

1. "Extreme Programming Explained" explains the importance of separating values from principles and practices.

2. "When Agile Gets Physical" unpacks why using agile software best practices are horrible for hardware. This is "duh" if you're a hardware person, but the book goes back to agile/lean first principles, and then builds up an approach for agile hardware.

It got me thinking that we often conflate values and practices. Blinding adopting Scrum or DevOps without understanding some of the underlying value and principles leads to problems. If you understand the foundational principles, you're in a much better position to adopt/reject certain practices for your given situation.

In reality, most organizations just adopt a few agile practices, while rejecting the basic values.

[1] https://rapidlearningcycles.com/when-agile-gets-physical/

[2] https://www.amazon.com/Extreme-Programming-Explained-Embrace...

Does anyone use a different term that doesn't imply working oneself to death?

This was a bad system.

To the extent that “DevOps” means anything, it’s that the spheres of capability and responsibility need to have tons of overlap: SWE’s need to be in the oncall loop and know how the filesystem works, and SRE’s need to be serious coders.

That’s it: everyone codes, everyone operates. We all automate where feasible.

Sidebar: What’s with the crazy surge in BuzzFeed-style titles on the front page? It’s not a federal matter but it’s not a slight change YoY either.

That said, I probably wouldn't want to hire someone with just one hat. It's ok to have a preferential hat, it's not ok to refuse to wear any other.

Then you have Ops person integrated in every dev team/squad, then those Ops people have their own meetings to talk about Ops things. Then there aren't enough Ops people to staff every dev team (maybe they are on vacation, perhaps just not enough in general), so they start sharing them. Next they need to work on some special Ops tasks, so they want to have their own board. Oops, now you have Ops team.

The point of DevOps is that the team is responsible for code being shipped, and after they shipped it, so everyone needs to know how to work in the stack that runs your app (Kubernetes, Nomad, AWS Beanstalk, bare-metal with Ansible or whatever it is you use). That means devs don't go to some other team to fix their CI or CD pipeline.

I worked in a company that was doing DevOps, we had no Ops team, we had a few people who only worked on infra, but they were part of our team. We didn't call it anything, it was just how to work, about 3/4 of the team was capable of setting up infrastructure and working on Jenkins.

I also worked in the company that "did DevOps" had a "DevOps team" and devs knew absolutely nothing about what happens to their code after they did `git push` (that if they knew how to do `git push` without asking "DevOps team").

Another company "for sure did DevOps right", they didn't have "DevOps team" instead they had "Infra team" and infra team was responsible for bare metal provisioning. Devs could write CI pipelines and stuff. However, devs had zero responsibility for code after they shipped it - it was infra team that would be woken up by nightly Pager Duty alert.

It's ok until the market decide it is no longer ok. So far, it looks like all these "hats" are complicated enough for the average to make the market think twice about demanding that combination of breadth and depth of skills.

* as a developer my free time should be spent on keeping up with my ecosystem (e.g. new JDK versions, Scala/Kotlin taking over Java, Akka-like modern libraries, the latest Flink features) and keeping an eye on possible alternatives (e.g. are golang generics good enough to make it competitive with Java?).

* once they standardize on a single DSL sign me up. I don't want to constantly waste my time on switching to CloudFormation, then to Terraform, then to Helm charts. Sometimes even moving to a different team in the same company requires it. Life is too short to be a full time YAML developer. Also, AWS CDK is the only option that at least resembles programming and it's apparently the least popular one.

It has nothing to do with frequency of deployments, and it does not mean that no ops team is necessary for running the show, it's about team autonomy and better understanding of different teams' concerns because everyone is doing a bit of everything.

> They believe in DevOps ideas, but they’re unwilling to invest in changing their ways. They aren’t convinced that some of the scarier ideas (like deploying multiple times per day) will help after all.

Option number 3 is the biggest one for me: thinking that having a separate "DevOps" team from the Dev team that does ops via modern tools is DevOps. That may be a good approach for your company, but it ain't DevOps.

I've come across this archetype in several contracting gigs and I find it to be especially pernicious as it gives the incumbents (who usually have more weight in the org) ammo to effectively do nothing.

And if you aren't willing to trust your delivery teams then your organization has a much bigger problem than either DevOps or Agile can solve.

It is like saying that C programs never crash, because if they crash, it means you are not using C properly. It is true, but no one will pretend that, everyone knows that under real life circumstances, C programs can crash, and there are plenty of other programming languages that address this issue, at the expense of other things, like performance or simplicity.

But for some reason those who push methodologies don't seem to consider that if that methodology is difficult to implement, or if it is not robust against things like conflicts of interest and office politics, or whatever, then maybe it is a problem in their methodology. It doesn't mean the methodology is bad, just that it doesn't work in every case, and that some tweaking may be necessary.

Want to save in project management? Just make devs responsible for planning and estimating, and call it "agile".

Want to remove sysadmins and ops? Make devs write their own configuration for the infrastructure, and say you do "devops".

Want to eliminate QA? Say that you do TDD.

I don't think anyone ever thought of it like this. Well, some dumb managers probably did. If you want to eliminate QA people, then you need an enormous automated E2E test suite. To have that, you need dedicated people who think of and write those test...one would call it a QA team.

When I point to the moon with my finger and say "moon", the fool looks at my finger and calls it the moon.

Is it more important to mindlessly tag yourself "Agile", with all the bells and whistles adorning the capital A, or would you rather just be agile according to the sought out goals and philosophy, and adapted to your specific situation?

Is it more important to have a clearer definition of "DevOps", so that you can more accurately implement all the prescribed workflows, or would you rather understand the pursued ideals, so that you can pick and choose what you need, according to your specific context, unencumbered by a slew of best, but ultimately useless, practices?

I appreciate the alert that there's a problem here, but there's no substance to the article.