Senior leaders and people that want to get there take note: cognitive load is the base problem one solves for when scaling development organizations. This article is a pretty good introduction -- especially building with empathy and emphasizing "what" over "how". But one can also look to any tech company that provides an API devs like (my fav: Stripe) as an example of what low cognitive load relative to the problem looks like.
Truly talented leaders tend to realize that high cognitive load comes from lots of places besides technical considerations too. It can be hard to think about a problem when you're also dealing with a toxic team member, untrustworthy leadership, lack of organizational focus, shitty HR policies, feeling unsafe at work, etc. Unfortunately, fighting against those elements is a never-ending battle.
Leaders that mix low cognitive load with clear direction and an interesting problem start to approach that highly-sought-after "early startup productivity" that so many companies can't seem to figure out. At least until a re-org, acquisition, or change in the c-suite comes along and blows it all up.
> cognitive load is the base problem one solves for when scaling development organizations.
Just to add, this generalizes beyond just development orgs.
I've scaled out analytics orgs at a few companies, and one of the most difficult aspects is penetrating into the workflows of business functions. As a rule, most[1] business users tend to operate at a high level of cognitive load, with little in the way of support structures to reduce that. But effectively integrating operational analytics requires[2] contextualization. So more often than not, the analytics scaling acts as a forcing function to put in place the operational and process supports needed for that contextualization, which has the secondary effect of reducing the cognitive load of the end users. In more than one occasion, I've seen that secondary effect have a greater impact than the analytics itself.
[1] A major exception to this rule are tech companies and companies with technical founders. If the core of your business is a technical system, then your business functions are created as extensions of that system and you get a lot of process/structure "for free" on the business side due to that fact. And if that's not the case but you still have technical founders, those individuals will approach business functions with a process/systems-first mentality since that's how they approach technical problems. While it has it's own issues, it does lend itself to reducing the cognitive load of those functions and enabling more efficient scaling. Business leaders will take the opposite approach, where they leave the function mostly unstructured and (ideally) create process, structure, and systems based off of the emergent needs.
[2] You can certainly provide analytics deliverables without contextualization. But it won't be nearly as effective, will mostly be ignored, and will rarely actually help anything since it leaves the onus of contextualization and interpretation on the end user and therefore does nothing to reduce their cognitive load.
Any particular industry/function/scenario you'd like used as an example? I can think of plenty of examples, but not sure which would click the best for you.
+1 for an example - not for the "major exception" but rather for the "rule". I'd love to find out the specifics of contextualization that you mention - how should it be done, what's the ideal process for it, to deliver a valuable analytics deployment in the end that won't get buried due to lack of use several months later.
Smaller business teams tend to use the "shared hosting" model when it comes to staffing: they hire a person and start subscribing relevant duties/needs to that person. At some point in time, they'll get oversubscribed to the point where it noticeably impacts their latency/performance. Once that happens consistently enough, the team will (hopefully) hire a second body and overflow duties to that new person. Rinse and repeat.
Think of a 1-2 person marketing team, that has to handle anything that crops up and happens to fall under the "marketing" umbrella.
At some point, demand for particular workloads will reach a point to justify adding in a dedicated server or two (i.e. hiring a specialist).
But, while you now have a specialized/dedicated resource to direct specific types of work to, it's still a "subscribe relevant duties" reactionary model. And the specialized resource gets utilized/subscribed to just like a general resource would, but only for specific types of work.
The day-to-day for these people is to execute work in the same manner as an interpreted language - they parse the request, constraints, and their capabilities, then immediately execute that work. Very little in the way of formal or documented processes, structures, outputs, inputs, etc exist. While some processes might be "cached" if they occur frequently and consistently enough, at some point they'll be garbage collected and have to be re-interpreted next time they execute. This is the reason most business teams operate with high cognitive load - disparate workloads, assigned with an oversubscription model, that require going through an interpretation phase before every execution.
Some particular workloads may be consistent or important enough to warrant a JIT compilation model. Or they'll be underutilized enough to have the capacity to proactively implement a JIT compilation model. Components of the work will be precompiled and defined and structured, then at runtime they can reduce the interpreter overhead (cognitive load) necessary for execution. Very, very little of their workload ever makes it to a state of ahead of time compilation and the corresponding reduction in runtime overhead.
That's the general rule for business teams. Developers though are so expensive that they're rarely allowed to operate this way. Instead, it's (theoretically) more cost efficient to invest in JIT optimizations for their workloads in the form of BAs, PMs, DevOps, ticketing systems, formalized processes, etc. And because they create systems and structure for a living, they're more apt and able to precompile parts of their workload themselves far more efficiently and effectively than a business team can do.
As for ensuring a valuable analytics deployment: there's no single answer to that, it has to be interpreted at runtime. ;)
But as a generalization, focus your analytics on giving your end users the tools necessary to be better JIT compilers. And ensure you don't introduce runtime errors for them. If your analytics deliverable is relevant to 80% of process executions, and has a known 20% of the time that it's not applicable, the user will lean into it and rely on it as a runtime optimization when appropriate. If it doesn't work for 20% of executions, but that 20% isn't known ahead of time, it'll be seen as a potential risk of runtime errors and completely rejected or bypassed by the end user. And if you prematurely go past that and leverage analytics for AOT compilation (i.e. full automation), your deployment is likely to fail entirely due to adding in too much rigidity to accommodate the unstable and ill-defined state of the process itself. Or if you go to the other extreme and your analytics just provides more inputs at interpretation time with little benefit of JIT improvements, it's just more runtime overhead and will have limited adoption unless the end user is forced or has enough spare capacity to absorb the additional interpretation overhead.
Feel free to reach out if you want something more contextual - my email is in my profile.
Just also wanted to add a big factor can be personal problems as well. Many of them can't be helped, but things like transportation, commute time, local real estate prices, health insurance, etc. can be helped and should not be overlooked.
Unfortunately not any that still exist. The only companies that seem to be able to accomplish it are early stage startups; because it is orders of magnitude easier to do with such a small group of people. Code School & Envy Labs are my personal examples.
I do think it is possible at large companies. I've built, been part of, and talked with managers that have created teams that obviously have low load (due to how much they produce, satisfaction with work, and level of trust). But it never lasts for long. Inevitably something like Radford is implemented by HR or some VP gets the reorg itch and tears down the system because these sorts of details haven't made it into boardrooms beyond "two pizza teams". Pluralsight and Amazon are my personal experiences with large companies.
This is why people who can't handle too much cognitive load end up being better leaders.
Or in other words... less intelligent people make better leaders.
On a side note, less intelligent people also write more readable code. The principle, Keep it simple stupid, aka kiss is indeed better followed by people who are described by the acronym. So in other words... A simple and stupid person is better at keeping their code and designs simple and stupid than a smart and complicated individual.
I respectfully disagree. Simple and elegant code is extremely difficult to write, and the ability to do so is mostly orthogonal to intelligence. However, below a certain threshold of skill and capacity for reasoning about spatial complexity, the developer is much more likely to write confusing, tangled, unnecessary, and verbose code. Reading through a codebase with tons of copy pasted code requires far more effort than a codebase with well designed abstractions. The key phrase there is well designed. What you are describing is someone who know which abstractions to use when and where, and that is a skill that requires lots of experience and technical maturity.
Experience and technical maturity do not equate with intelligence.
Abstractions serve one purpose and one purpose only: to reduce cognitive work load. Any abstraction above a primitive implementation only can offer added inefficiencies . Ex: SQL is less efficient than C++ which is less efficient than assembly. A zero abstraction code base is the usually the most efficient implementation and it will be written in assembly.
So from a technical standpoint, we use abstractions only to reduce cognitive workload because other than that abstractions can only offer inn-efficiencies.
Intelligent people do not put in the effort to learn about or implement proper abstractions because they usually deem it unnecessary to abstract what they perceive to be trivial cognitive workloads..
I respectfully disagree with your disagreement of that disagreement.
Abstractions have far more roles than reducing cognitive load. You're completely overlooking platform realities, code reuse, usability and other benefits of reduced/managed complexity.
From a technical standpoint efficiency of the code is irrelevant if it's bug-ridden due to it's complexity. Code is for humans, not the other way around. We chisel away at lower-level languages if efficiency is required (Ex. C bindings in Python).
If you want to utilize your intelligence to the fullest you abstract away most of the trivial stuff to the point where it pays. You can still go down and inspect or override the abstraction if it's needed. Experience in this case is knowing what to abstract in what manner so it will work for you. Intelligence is the act of keeping everything in a sane state without over-focusing on unimportant stuff.
What I think you're critiquing is the act of adding abstractions when there's no need for one at a given time, just to make something simpler in name of simplicity overlooking it's usability. This can be attributed to the lack of experience.
KISS is a suggestion, not a rule. It also applies to abstractions, so in one could argue that doing everything in assembly is actually the "simplest" way of programming, like a rough sketch of a scene is simpler than a full-blown oil painting.
As an aside the whole notion of "intelligence" is a bit twisted with experience IMO. The "classical" IQ applies mostly to dumb pattern matching - a skill one can perfect. EQ can be trained by getting out and deliberately practicing human interactions.
I respectfully disagree with your disagreement of that disagreement of that disagreement.
>Abstractions have far more roles than reducing cognitive load. You're completely overlooking platform realities, code reuse, usability and other benefits of reduced/managed complexity.
I am not overlooking anything. The traits you bring up in this statement do offer any performance improvements to the system. Therefore the only other possible benefits that these traits offer is that they reduce cognitive overhead.
>From a technical standpoint efficiency of the code is irrelevant if it's bug-ridden due to it's complexity. Code is for humans, not the other way around. We chisel away at lower-level languages if efficiency is required (Ex. C bindings in Python).
Yes. And intelligent people can write more complex code with less abstractions and have less bugs... We agree.
>What I think you're critiquing is the act of adding abstractions when there's no need for one at a given time, just to make something simpler in name of simplicity overlooking it's usability. This can be attributed to the lack of experience.
I am not critiquing anything. I have not said anything about my opinion on where or when to add abstractions. I have only commented on how a very intelligent person would do it. I never said I was intelligent... All I said was more intelligent people tend to write less readable code and this can be attributed to the fact that they have less need for abstractions. So no I am not critiquing about when or where to write abstractions.
>The "classical" IQ applies mostly to dumb pattern matching
IQ tests present questions with patterns that the test taker usually has not seen before. A test taker cannot "dumb pattern match" a pattern he has not seen. Therefore the IQ test cannot be testing for "dumb pattern matching." If the IQ test measures IQ and the IQ test is not measuring for "dumb pattern matching" then by concrete logic IQ must not apply to "dumb pattern matching." QED
IQ must apply to something more. A general intelligence.
If you're working with what you perceive to be "trivial cognitive workloads" I think that is more a sign that you aren't quite working at the level you're capable of, than a sign that you're a really intelligent person.
The thing is that all humans have a limited working memory (which generally correlates with 'intelligence'). Whether you can juggle 3 objects in your working memory or 30 doesn't matter; eventually you will get to a problem that you can't hold within your working memory. At that point you will need to start using abstractions for both object-structure and functional characteristics. The limit of what you can do with abstraction is way higher than that of your working memory or intelligence.
As a rule it's always better to properly abstract and structure your code, to keep it as simple and well-formed as possible, and to let your tooling deal with any higher-order actions.
As a bonus this allows other people to have an easy time understanding your code, especially when those 'other people' are your future selves.
>If you're working with what you perceive to be "trivial cognitive workloads" I think that is more a sign that you aren't quite working at the level you're capable of, than a sign that you're a really intelligent person.
Logically it can be a sign of either. If the cognitive workloads are on average hard for normal people and you perceive it to be trivial than you are likely intelligent. If the workload is considered to be trivial by most people and you see it as hard than you are likely not intelligent.
>The thing is that all humans have a limited working memory (which generally correlates with 'intelligence'). Whether you can juggle 3 objects in your working memory or 30 doesn't matter; eventually you will get to a problem that you can't hold within your working memory. At that point you will need to start using abstractions for both object-structure and functional characteristics. The limit of what you can do with abstraction is way higher than that of your working memory or intelligence.
I get where you're coming from. Think of it this way. If we organize the complexity of programs into layers such that each layer functions as an abstraction to reduce cognitive overhead, then we can use the number of layers as a quantitative measure of how much abstractions were used on a program.
A very intelligent person would use 3 layers while a less intelligent person would organize the program into 10 layers. There see how it works? Yes complexity is potentially unlimited but the more intelligent person still ends up relying on abstractions less than the unintelligent one.
One thing that does equate with intelligence is capability for metacognition.
Less intelligent are also those less capable of realising that they’re overburdening their cognitive capacities with their convoluted code.
More intelligent people are likelier to notice that they’re making things harder for themselves (even if it is within the realm of tolerability) and look for alternative strategies.
I actually agree with this. I've seen it happen but not in the way you expect.
I've seen less intelligent people write unnecessary design patterns and abstractions and end up making things harder for themselves. In essence, the heavy reliance the less intelligent programmer has on writing hyper readable and abstract code ends up damaging the app overall.
> Ex: SQL is less efficient than C++ which is less efficient than assembly. A zero abstraction code base is the usually the most efficient implementation and it will be written in assembly.
I don't think we can compare SQL and C++ like that. Not at all.
SQL implementations are most likely written in C or some systems level language. It is a language built on top of another language. A higher level language built on a lower level one. It is therefore by definition a higher abstraction and thus since the nature of the comparison is based off abstractions the comparison is apt since it fits the definition of the word. What you think is different then facts defined by words, and therefore your thinking: that you can't compare C++ to SQL falls in the face of facts.
Perhaps for a very narrow definition of efficiency. Writing in assembly what could be written in SQL is definitely less efficient from a productivity point of view. Even for your definition of efficiency that may not be true.
>Writing in assembly what could be written in SQL is definitely less efficient from a productivity point of view.
You mean productivity as in cognitive workloads can be so high in assembly that it's not efficient? I choose to use a narrow definition of assembly because my argument is that the productivity gains given to an intelligent person by abstractions above assembly language are inconsequential. It wouldn't make sense to use a broader definition. You use the word Productivity but we're really talking about the same thing or in other words "the loss of productivity due to cognitive workloads." Thus your argument is circular to mine and therefore ineffective.
My definition of efficiency is true. SQL eventually executes assembly commands thus it is a higher level of abstraction than assembly and thus it can only be equally or less efficient in terms of space and time complexity. Realistically, the assembly code actually executed is most likely not the most efficient way to execute the code.
I can only assume that you were trying to say that the more intelligent leaders are worse because they cannot understand more complex systems as easily and therefore require things to be done with lower cognitive load.
Regardless of how simple your delivered system ends up being, you still have to understand what problems you're solving and the reasons behind it. This is almost always a product of factors outside of your control.
Let's assume that in the worst case, you're going to be facing a very complex situation. Now you have three possible outcomes here when delivering.
A) Your solution is simple, because you either did not grasp (or were otherwise unaware of) the complexity of the problem, did not acknowledge it, or disregarded the severity of the problem. Or your organization's engineering culture puts pressure on you to cut corners (e.g. "Move Fast and Break Things").
B) Your solution addresses the complexity of the problem, but it was a lot of work to solve and you did not have the time or experience needed to build it in an elegant or optimally maintainable way.
C) Your solution addresses the complexity of the problem, and is built in a reasonably elegant, simple manner.
Which of these did you say correlates with intelligence again?
Edit: There's also a fourth outcome, reserved for the real rockstars out there.
D) Your solution is to not solve the problem at all, because you found a way to avoid your organization/product having to deal with the problem to begin with, or you otherwise re-assessed the assumptions surrounding the problem. The famous post from a few years ago "Why I Strive to be a 0.1x Engineer" comes to mind.
Good leaders tend to use the intelligence of their team to determine direction and design. They have meetings and empower team members to perform at 110% by giving the team members ownership and clearing away obstacles.
Less intelligent people are better at this because they have no choice but to do this. Less intelligent people do not have the cognitive power to design the full system so they have to resort to relying on the team to do it.
Very intelligent people tend not to do this. They rely on their own intelligence to direct the team and often times just use the team members as grunts to implement there grand designs.
No matter how smart you are, your cognitive powers will always be lower than the cognitive powers of the entire team combined, therefore any attempt to have the team follow your lead rather you following the lead of the team will be worse off.
The science supports this. Teams with leaders that empower team members are statistically more successful than teams that don't. My anecdotal observation is that these types of leaders tend to be the less intelligent types. They empower you and than get out of your way because they really don't have the intelligence to get in your way.
Here are two statistics about humanity that will lend support to my thesis. Your IQ does not correlate with how successful you will be in life, success being measured as how high you climb the corporate ladder and how much money you make as an individual. This is an actual correlation. In direct contradiction to this... the average IQ of a country highly correlates with the economic success of that country. These two statistics say something about the nature of leadership. It says that leaders succeed off the backs of people more intelligent than them.
I'm not sure if I'm misreading something, but are you claiming that higher IQ does not correlate with higher income? You'll have to source that claim, if that's what you're claiming.
You are not misreading something. I made a mistake.
IQ does correlate with salary but that correlation is weak and a higher IQ brings in only a slightly higher salary. Correlations between IQ and GDPs however are very strong and a small increase in average IQ is correlated with a significant jump in GDP.
My source is the hive mind book, and my original footnote about IQ was incorrect and completely wrong. My original thesis still stands though. Do you have any thoughts about that? The whole IQ and success thing was just a footnote. I shouldn't of added it in, as you are correct in identifying that the footnote was completely wrong. I believe in admitting mistakes and not charging forward with bias so please critique anything you disagree with... I want critiques that address more than the footnote.
I read (between the lines, and maybe mistakenly) that you consider yourself highly intelligent. With that perspective, the comment sounds like reasoning for why you aren't going to be a good leader and why that is a good thing. Also, it sounds like it might even be an excuse for writing unreadable code.
This perspective seems based on a premise of "intelligence" being a one-dimensional scale where everybody has an absolute ranking. I think this is generally a problem with the "IQ" concept that we've all agreed on. It seems extremely oversimplified to me, at least if we want intelligence to mean something like "brain capacity".
>I read (between the lines, and maybe mistakenly) that you consider yourself highly intelligent. With that perspective, the comment sounds like reasoning for why you aren't going to be a good leader and why that is a good thing. Also, it sounds like it might even be an excuse for writing unreadable code.
Humans tend to construct biased logic around themselves to justify their weakness. We lie to ourselves so that we may be happier. In fact, although there is currently a crisis in the veracity of psychology papers, I recall seeing studies showing a positive correlation between how realistic a person was and how depressed he was.
I can tell you that I strive to be as unbiased as possible. I strive to remain unbiased even when it hurts. This is not a trivial skill. I have not seen anyone call themselves stupid even when they clearly are, in fact the way we lie to ourselves is so powerful that even if you are stupid, you wouldn't know it.
That being said I would say I am more intelligent then average. I don't have IQ test scores but I have data that by common sense should correlate. I went to UCLA where the admit rate for engineering 11%. By that fact, I would assume I have above normal intelligence.
I actually believe the opposite to writing unreadable code. In fact I have a bias for readability and extreme abstractions to a fault. I am a huge fan of functional programming. My observations about super intelligent people are of others not of me (note that I only believe I am higher than the average, not someone incredibly smart).
I also believe that I know what it takes to be a good leader, that's why I'm able to identify the traits that make a bad leader and thus come up with this conclusion.
Please stop trying to equate my personal experience with what I am trying to say. I present my views and opinions in a way where I strive to be impersonal. Please judge what I say with the same impassion.
>This perspective seems based on a premise of "intelligence" being a one-dimensional scale where everybody has an absolute ranking. I think this is generally a problem with the "IQ" concept that we've all agreed on. It seems extremely oversimplified to me, at least if we want intelligence to mean something like "brain capacity".
Well clearly when you have opinions about people. In your vicarious opinions you and others often make a judgement that person A is more intelligent then person B. The ranking is based off of that fuzzy general opinion and my thesis is in no way an exact law or proposition. My thesis is just a fuzzy generality that I believe is true.
Those leaders are gifted with a lack of intelligence. People with lower IQ don't necessarily appear less intelligent. In fact they can be more charismatic and appear more intelligent.
Steve Jobs was not an intelligent person when compared to Woz. Yet Steve is often referred to as a genius. I don't think Woz could ever come up with the design ethos that Steve Jobs had around products and this is due in great part to Woz's higher level of general intelligence (IQ).
I don't know about anything that points to a statistical advantage. I'm sure you can find some small studies via google. Since statistical evidence is rigorous and hard to find, a good portion of the knowledge of an intelligent mind must be derived from induction and anecdotal evidence with lots of assumptions. The origins of my argument are anecdotal in nature. Statistical evidence is stronger but so hard to establish that if all my arguments are always based off of statistical evidence I would hardly have anything to argue.
Additionally statistical conclusions are already established. Hardly worth arguing about all together. It is the things that aren't backed by statistical evidence YET, that are more interesting and worth talking about.
Other way around. I consider myself as someone who writes extremely clean code. I consider many people who write messier code than me as people who are smarter. I self depreciate my own intellectual abilities in service of rationality. I write cleaner code because I am not intelligent enough to write messy code.
Honestly, I detect a thinly veiled personal ire in your response. Perhaps the bias is with you. Perhaps you write clean code and you use it to justify your supposed intelligence. If so, you wouldn't even know it. Such is the nature of bias.
Writing "clean code" is in actuality a simple endeavor with high euphoric rewards. All you're doing when writing "clean code" is following conventions and modularizing your code. It feels good to organize your room just like it feels good to organize your code, but organizing your room is not an intellectual feat just like how writing clean code isn't either. Think about it, cleaning up your own code is really, really easy. To have it function as the pillar of your intellectual self esteem is a weakness.
So I haven't found any of the common measures of intelligence (self-reported IQ, presence in honors/accelerated programs in school, "seems to learn new concepts quickly," etc.) to be indicative of anything particularly useful in a work environment. So how are you measuring intelligence to make these sorts of statements?
I ask because reading your comment leaves me with the impression that there's some conflation between intelligence & empathy + humility going on.
> So I haven't found any of the common measures of intelligence (self-reported IQ, presence in honors/accelerated programs in school, "seems to learn new concepts quickly," etc.) to be indicative of anything particularly useful in a work environment.
Right and this is in line with my statements. Less readable code by common sense will not correlate with being more useful in a work environment. Though intelligent people tend to have other benefits that allow them to excel.
>So how are you measuring intelligence to make these sorts of statements?
Anecdotal evidence. I try to be as unbiased as a biased creature can possibly be.
>I ask because reading your comment leaves me with the impression that there's some conflation between intelligence & empathy + humility going on.
I don't know where you get this from. All I am saying is intelligent people write less readable code... or vice versa That's it. I have not commented on any other factors.
It isn't bad per se but it needs to be applied strategically. After all, this ability is exactly what enables some people to deal with problems of high essential complexity, achieving what no one else can. You just have to realize that the majority of software does not fall in this category.
It seems to be exactly the opposite instead: “Any fool can write code that a computer can understand. Good programmers write code that humans can understand.”
Truly talented leaders tend to realize that high cognitive load comes from lots of places besides technical considerations too. It can be hard to think about a problem when you're also dealing with a toxic team member, untrustworthy leadership, lack of organizational focus, shitty HR policies, feeling unsafe at work, etc. Unfortunately, fighting against those elements is a never-ending battle.
Leaders that mix low cognitive load with clear direction and an interesting problem start to approach that highly-sought-after "early startup productivity" that so many companies can't seem to figure out. At least until a re-org, acquisition, or change in the c-suite comes along and blows it all up.