Earlier this morning I spent 45 minutes in a meeting discussing the replacement for a $4 million big data system we bought 2 years ago because no one was using it. The solution discussed wasnt to ask the analysts why no one was using it but to spend $20k on POCs for two competing systems each costing a further $5mil as their on-paper specs are marginally superior.
My head is nodding at the intention here, but it's important to remember this reactionary attitude can be damaging as well. That is missing from the post.
Specifically, reactionary conservatism about tools and services can be a competitive disadvantage.
In several workplaces I have seen this attitude co-mingle with pride in original code, and result in a counter-productive NIH (Not Invented Here)[0] attitude (reinventing wheels). Macho/heroism makes it worse.
While trend-following can be inefficient, if the engineers are both excited and humble and ready to weigh many trade-offs, this inefficiency might be "cheaper" than reinventing wheels for all problems.
To clarify I'm talking especially about _uninteresting problems,_ for which there may be an obvious off-the-shelf solution, such as when I've seen engineers invent a build system from scratch (and then cost _lots_ over years as that custom thing needs maintenance) ... when their needs could have been met by the language's standard tooling and using its plugin architecture.
Or, when "just use sticky notes and a whiteboard" gets so stubborn that a growing team stays in chaos for lack of organization.
On the other hand, it is quite bad when trend-following causes not just churn (switching tools), but overkill that introduces tons of cost and risk. i.e. distributed systems where you didn't need one. Relevant on this point: You Are Not Google by Ozan Onay https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
The point of the text is that a need, and an understanding precede the use of any tools. The use of a tool without a need or an understanding does not help at best, and is harmful pretty often.
You do not become a photographer by buying a lot of gear. You train your eye, study, and maybe buy gear eventually when you know why you need it.
You do not become a software developer buy buying a bunch of supercomputers. You study, you start with smallest things, and maybe you buy hardware eventually, or deploy a lot of cloud instances, when you know why you need it.
You do not become a successful company by quickly raising tons of capital. You study, learn about the markets and the needs of your prospective customers, start with building rather simple things, and maybe eventually you take hefty investments, when you know why you need it.
You can go on and on. The old adage is to use the right tool for the job. This begins with understanding the job, and understanding the need of tools. Then you can choose the tools. Starting with tools, especially expensive tools, and believing that the use of these tools will automatically help achieve success, is a mistake.
Good point. The piece is sound in the abstract and is good advice to recall when you consider the essence of your endeavors, but tooling does count when you consider the material situation—some approaches become untenable at certain magnitudes. Funily enough, the best we can do is often trade a big problem for a smaller one—« how do we serve X million people » becomes « how do we teach x thousand employees to use the tool that enables us to serve X million people «
It's fair to say you can't process 40 million bits of data on a whiteboard. On the other hand, if you have 40 million bits of data, but can't work out how to answer the question you're asking of it (or in many cases, what question you're trying to ask!), then it's useless.
This is where whiteboards, paper, and other physical solutions really shine. You can make notes for yourself or communicate with those in your shared physical space in a very immediate, speed-of-thought way.
In general, software (even BI software) is limited to doing what it is programmed to do. We necessarily adapt our thinking to the tools we have to express the ideas we have. Dancing about architecture is difficult, as Steve Martin pointed out.
Well yeah, but before you invest millions in a BI tool, you should decide what you want to do with it first - you can work out what data and statistics you'd like to have in a smaller scale on a whiteboard first.
Investing in a tool without first doing the task by hand is wasteful. No I'm not saying you should do it with all 40 million subscribers, just start with getting one or two and a whiteboard, then a few dozen or hundred in an excel sheet.
If you charge enough, maybe I can get you to come in and consult with my bosses. I've been saying that for years and they keep mixing up the order. Now they've automated the wrong processes using custom in-house tooling and don't know why they're getting strange numbers (people fake the data to fit the tool, but do something totally different).
I wish it were that easy. But unfortunately, if the bosses aren't already asking these questions, they probably can't be consulted into it.
They might bring a consultant in, and nod away at his analysis. They may write some memos, talk about it in the manager meetings. But that's it. Nothing changes.
Management defines culture. Culture is a function of personality. Personality rarely changes. That is why leadership matters so much. That is why we have seen companies stuck in a rut for years, but then radically transformed when a new CEO is brought in. Or vice versa, companies that are doing well, and then tank under new leadership.
But hey, if you can talk them into flying a middle-market CIO in to give them a come-to-jesus speech, I'll be happy to take the money.
Good process is all about ensuring that expectations are clear and shared across the company. It does not meant that there's no room for judgement or that it treats every employee as a mindless automaton.
Everyone follows a process - the only difference is that if you write it down then you have a hope of everyone following a similar process. If you don't write it down then everyone just follows their own slightly different process.
This doesn't even touch on why process is good for scaling a company. It's easy to have everyone work to a standard process that isn't written down when there's five developers that sit in a room together. But at a certain point you're going to have to hire new employees and you'll need to communicate to them what the expectations are. You'll eventually have to hire "cogs" who aren't rockstars who just know what to do. You'll have people working in different offices and across timezones. Again, these people will all follow A process. By writing it down you can help ensure that they follow the process you want them to follow.
Ive seen many scaling issues so far as well.
However I tend to think that this is a problem with coherence in discipline of the hierarchy. Person A always breaking the build now just becomes Team A breaking the build.
So even still why not simply just use process despite its drawbacks of rigidity since it will still at least solve your problems.
The answer is that just like corruption no finite set of laws will ever be enough to curb an underlying behavior.
This is why things like mandatory code reviews are pointless unless people take them seriously
For small groups there is little difference between firing a person for not following cultural and authoritative norms vs constructing process around those and then firing when those are violated. The written process is merely a concrete representation for what is already known by the nominal parties.
Yes, my experience is very different than yours. But I have seen companies of all sorts.
First of all, lets define good people: Happy employees, who are engaged with their work, have the skills and the tools to do their job, and have a management team invested in their success. These are not clichés.
Very broken companies have bad people and bad process controls, or no documented process at all. If such a company is small, it may last for a while, but it is inevitably doomed.
Slightly better companies have bad people and good processes. By that, I mean they have reasonable process controls and accountability, but no program in place to ensure that they have the right butts in the right seats, and nothing in place to keep their employees happy and engaged. They are proactive as regards to process, but reactive as regards to people. This is probably the center of the bell curve.
Better companies have programs and hiring practices to ensure that they have and keep good people. But they may not yet have well-defined and enforced processes. If the company is small, they will probably do well. Process controls often come with growth.
Better still are companies have good people and well defined and enforced processes. But they don't have a culture of continuous improvement which is always looking for ways to improve, be more efficient, etc.
But the best companies of all have good people, and good process, but are constantly working to keep their people happy and engaged, and are constantly optimizing their processes to do more work for the same effort. Such companies are not exactly unicorns, but they are not common either.
The thing is, every organization has processes. Most just don't write them down (or what they wrote down isn't what they do).
When someone checks code into your master branch, do you let anyone do it? Do you do peer reviews (of some sort)? Do you first run tests? Those are your process. If you don't record them (via documentation or automation) then they're learned by word-of-mouth and can easily be miscommunicated.
Do you have a process (and this is more critical for larger organizations) for purchasing? A lot of tracking here has to do with legal and regulatory compliance, not just tracking for the sake of tracking (at least, for the company, the things being complied with may be worthless to you). Is this written down anywhere? Is it automated in some fashion? Who would you contact if you need to buy a new <something> for your office? What if you need more office space, how would you go about getting it? How does your office track reimbursement for work travel?
I agree that there there is an informal process that we call the culture. The smaller the org the less explicit process required. I dont know about "purchasing" but I do know software so let me address that.
The good SEs that I have worked with knew when to get reviews and when to run what tests. The good SEs tried their best not to harm others and knew when they were exceeding their bounds of authority. This was communicated via cultural transfer and mentorship.
The reason process tends to be bad is that tends to be cumbersome, inhuman, and ridged.
"The first value in the Agile Manifesto is “Individuals and interactions over processes and tools.""
There's process and Process. Big-P process is broken, it's too rigid and turns into quicksand. Or it's not what you do.
But processes happen everywhere, and even good developers aren't going to be uniform in what they do or have experienced. This is why I still find myself teaching 20-year programming veterans about make or *nix, because they've never used it in the past. With many of our more recent projects shifting to git (versus SVN or TFS for our historical projects), we've also introduced new workflows that people aren't used to. We can either have dozens of inconistently styled branches on the main repo, or we can establish a protocol/workflow. Once it's established we have to communicate it. The choice is: by "culture", which is loose, ad hoc, and often ineffective as people move in and out of teams; by process, document what that workflow is.
The follow up to that: If that process becomes Process (seemingly fixed for all time), you've lost. You should reexamine your processes as time goes on to make sure they're still correct, or that they actually accomplish what you wanted. Maybe a part of the workflow was to deal with a particular technological limitation that's since been removed, then change that workflow. But letting it go by word-of-mouth is not going to scale beyond small teams and small projects.
I totally agree about process vs Process. And I dont know who would every really be to opposed to process as guides or how tos. I am a huge fan of self-process and having my own coherent Confluence pages on how to do things and setups and FAQs.
The only thing I would say is that the word of mouth can work quite well still even in large spaces due to hierarchy. If someone on your team doesnt know how to setup a repo and there is no process It is likely the leads responsibility to convey this information.
Of course. But by "process," I think we all understand "defined process." If it's not defined and enforced, it's left to whim and chance. I wouldn't really call that process.
Because automation is time consuming! Process comes first because it's easier to fine tune the process, then automate it. If you automate too soon you waste time building the wrong things.
I could literally rewrite your comment swapping "process" for "people" and it would resonate with others.
People, process, technology. They are 3 pillars. You cannot build without all 3 working in sync.
Good employees can only do the right things with the right technologies - which may be a whiteboard and a notepad - and the right processes so that they know what the "right thing" is.
And you are absolutely right. It always starts with good people.
My company has good people. We have defined programs in place to make sure we have good people, or that the people we have are in the right seats and are engaged in their work. When we have toxic players who cannot be happy and engaged no matter where we put them, we move them out.
But we also had rapid growth, and so with good people came a scattered process - every project team did things differently. It has been steadily improving, and we are building the tools to make the process as effortless as possible. But before we did that, we spent months mapping out existing processes and standardizing them. It was painful. But it's also now fueling growth. More work gets done with the same amount of effort.
Management here tried to throw tools at it early on. It's the natural reaction. But thankfully, they learned quickly. We have a good management team.
I like this. I've gone really far with just Google Query or a simple SQL query.
Another idea I like is not to "fall in love" with your analysis, per se. I've seen plenty of extrapolations and wonky charts that end up just being a waste of time.
I think maybe there's some nuance here to how to think about tooling. A data science team without any sort of data warehouse is probably going to be ineffective.
Maybe one way to think about it: some tools try to model and streamline the things you're already doing, and some tools expand the set of things you're able to do. I think the first type is the one to be more skeptical of. Face-to-face conversations, whiteboards, and Excel sheets are all extremely flexible and powerful. It's possible for tools to improve on these, but it's a high bar.
I agree in principle but I've first hand seen what happens when a company that has grown doesn't get the tools it needs.
A ticketing system, for example, is how you create a unified view of what matters and what should be worked on. Otherwise it becomes a game of politics and social cunning down to the IC level which is not how I want to spend 8 hours of my day.
DAMMIT THEY WILL BE MADE TO READ THIS LINK