John Cleese has given talks throughout the years about how to be creative, and they're very good. [2]

The Programmer's Stone is also a good source of information about this topic. Instead of "lateral thinking", the authors refer to "mapping" and "packing". Packing is when you're in "step-by-step" mode. When programmers refer to "being in the flow", they're probably referring to the state of mapping or lateral thinking. [3]

[1] https://en.wikipedia.org/wiki/Lateral_thinking

[2] https://www.youtube.com/watch?v=y70nbDJI5Uk

[3] http://programmersstone.com/

To not allow your imagination to run ahead of or independently of your behavior, to not allow yourself any mental play, is to be unimaginative. That's tautological. If you truly find yourself unable to imagine any outrageous ideas, you have to look elsewhere than pragmatism if seeking a cure. You need to rediscover how to play. In your head. Pragmatism is a filter to decide when to put some of your play ideas into practice, not a filter for thought itself.

There are different challenges someone else might face, requiring different analysis. What if you are able to come up with outrageous and lateral ideas, but are afraid to voice them? Or you are unable to ground the ideas in any kind of reality, i.e. pure fantasy rather than plausible-if-ridiculous? Or do you find challenges in the social environment of getting any air time for your ideas among your peers...?

Perhaps it's Elon Musk's chats on thinking in first principles, but I'd love if anyone knows some introductions to linear, tight-knit, strategic thinking.

Edit: I also wonder if books on clarity in writing like Zissner's "On Writing Well" qualifies here. Writing is linear and the practice of it presumably should help the writer think more clearly.

In my experience, I actually think it can -- but you can overcome it, too.

I grew up programming and always excelling at concrete, black-and-white, logic-based types of things (math, physics, etc.).

Later I wanted to pursue more artistic things, and discovered I simply didn't know how to think imaginatively, instead of analytically.

After a while, I learned how to "let go" of my analytical brain so that my artistic/subconscious brain could be free to "make connections" between things that had no logical connection on the surface.

And now I have no problem with it -- so it's never "too late" -- but they're definitely two "modes" of my brain I have to consciously switch between. And they're largely "separate" modes that both have to be developed/strengthened through practice.

If you want to work on imagination, I highly recommend things like writing classes, improv classes, acting classes, painting classes, all that stuff. I think classes with other people in this context are really important, because the instructor and other students' work will give you a lot of examples of creativity to inspire you and build you up. If you stick solo (as I did at first) you may find yourself just applying the same "analytical" solutions you're familiar with, and not being forced to expand your imagination/creativity.

This is a blocker. One of the ways to unblock this, I'm told, is the "improv" mantra of "yes, and" -- meaning, if someone states an idea, instead of saying "yes, but", you say "yes, and" and build upon it. For instance, someone says "I'm a unicorn fireman", you say, "yes, and your horn is a nozzle for a high-pressure hose". And you keep building.

For years I could never quite figure out how this applied to engineering problems so I thought "yes, and" was mostly stupid and unworkable (except in the arts) -- because what if the other person's wrong? How do I "yes, and" someone who says "the earth is flat"?

But one of Paul Graham's tweets helped me reframe this in a way that is palatable to even engineer-types. He says (paraphrased):

"Problem solving is 2 phase approach: (1) idea generation / mistake making, and then (2) aggressively fixing those mistakes / editing. We often shortcut the process by editing too soon."

In other words: engineers are only good at "yes, but" (convergent thinking phase) and prematurely shut down the "yes, and" (divergent thinking phase). So how about we treat thinking as a two-phase problem and give divergent thinking some room? We can still apply pragmatism/practicality ... but later, after we've "yes, and"ed.

Alls we're saying is, give "yes, and" a chance.

I do this sometimes. In many cases, I feel like "yes, and" would be an endorsement of the original idea. If I say "yes, and" and then there are delays or problems in implementation because of things that would have been obvious to "no, but", the question becomes "why wasn't this brought up earlier on before people sunk so much time in." In the reverse case, saying "no, but" in a case where the problem is easily surmountable just makes the questioner look like a good planner. So in that sense the reason devs tend to "no, but" is matter of them responding to incentives; it's not that they can't think divergently, it's that they're putting themselves at risk for little reward for doing so.

> So how about we treat thinking as a two-phase problem and give divergent thinking some room?

Yes, I think separating these two phases (and thus separating "throwing an idea out there" from "endorsing an idea as definitely workable with no foreseeable complications") would be very helpful.

Being primarily an engineer myself, I can't help to say 'yes, but' you might want to rephrase this into 'untrained engineers' or 'some engineers' or 'uncreative engineers' or similar. Don't get me wrong, I know the people you talk about and I simply cannot deny I used to be like this myself sometimes but over the years this changed into something bimodal. I.e. when needed I will always fall back to 'how can this fail' mode - just because it's often required to properly fix/create things which won't brake, but I find myself defaulting to generating tons of ideas. But I'm also creative b nature so that helps. In fact most other engineers (by education or not, doesn't matter) I know also clearly use a mix of both modes. There's only a couple which don't (and even some which default to only creating a mess, not seeing any problem, but that's something else and I'm not sure if it's an engineer:).

yes, and "resting on the backs of four huge elephants which are in turn standing on the back of an enormous turtle"

I'm engineer :)

To use your "I'm a unicorn fireman" example, sure that conversation can be fun.. and what does the winner get? With that example you're not building a system to be used by humans in the real world, you're doing stream-of-consciousness thought experiments, more art than engineering. There are no solid constraints.

Sure there's some debatable evidence that creativity can be "practiced" and generally applied, but when you're trying to build a better airplane putting a giant swimming pool in the nosecone is probably not a good idea for a whole host of reasons and it's best that you account for those before building the thing.

Software and the other arts largely benefit from cheap iteration cycles and relatively low-impact mistakes. And even in those cases if you "yes and" too much and try to account for the mistakes later you can find yourself with a horrible, underthought design and swamped trying to fix all the problems at once. Then all those negative constraints you ignored in the name of "divergent thinking" suddenly make themselves known. Many a failed startup led by a Steve Jobs wannabe is a perfect example of this.

I think the main point is that there may be advantages to explicitly giving room to both divergent and convergent thinking to exist in separate phases (this could be done in one cycle -- there's nothing that says any of the cute unworkable ideas would ever leave the meeting room or shopfloor or prototype. Reality/logic/laws of nature are the final arbiters).

Constraints/refinements/criticisms will emerge in the second phase, but we often don't give room to the first because we tend to want to edit too soon, and if we're honest, our impulse criticisms going to be based on our past experience and emotion rather than a cool, deliberate, rational analysis process or empirical reality.

A disciplined balance is what is sought. In creative processes, withholding judgment during ideation is a basic principle -- but many engineering types are too quick to ignore it, wanting to filter almost immediately. This is ok if the space is well-understood and what you're doing is cookie cutter (which to be honest, is all most engineers do), but definitely a blocker if you really want to challenge ideas and do something truly new. It snuffs out ideas pre-maturely and doesn't allow them to potentially grow into something that would work.

An example of this: I was against neural networks for years because they lacked explainability ("black boxes" was my criticism of them), and whenever someone brought them up I would enumerate my usual list of criticisms of why they would never work -- all rational and mathematically watertight. It became so habitual that a few good ideas (where neural networks actually would have worked if we had tried them, albeit it had to be in creative ways) passed us by. I snuffed that idea out before it had a chance to emerge in a form that I would have recognized as something workable.

An example I experienced a couple of weeks ago: Someone let code that "just works" slide through a code review back in 2017. That code had a hard-coded value that would have been easy to determine dynamically (he just set a max at an arbitrary value instead of counting how many of a resource there were) but wasn't due to general lack of discipline. Now it's a minor issue, and the code did indeed work up until a couple of weeks ago. Then the number of the resource in question changed, and all of a sudden that hard-coded max was no longer accurate.

The error manifested as a segfault in our C++ but only sporadically and was hard to reproduce when we didn't know what the issue was. Doesn't help that it was in a mountain of legacy spaghetti code. It took me and two other people two days to track down the original cause, which if you add up our de-facto hourly rates cost the company thousands of dollars. All because of one little minor piece of sloppy code.

Now imagine the effects of bigger, grander sloppiness at the design level. And then those effects get compounded because they're conveniently ignored and integrated into still further mistakes, and you get an exponentially multiplying mess.

I'm willing to bet many professional engineers have had an experience similar to that, and if they care about their craft at all it was likely extremely frustrating and painful to deal with. So it's not that they're "bad at divergent thinking" so much as they've been trained by experience to believe that the benefits of divergent thinking aren't worth the potential downsides. Whether that's an unconscious bias that should be worked against is an interesting question.

But that superpower, which is so instrumental for execution, seems to me to be simultaneously deleterious to creative enterprises. It is at best an incomplete superpower.

I would also say that the reverse is true. Artsy creative types who are gifted at divergent thinking, but not convergent, are typically weak at execution so their products never gain traction or do not actually work.

Creative endeavors are by nature "sloppy". Hence the need to withhold judgment and let an idea play out via a divergent phase, and then bring it back through the convergent phase.

It would seem to me that both rationals and creatives have blind spots, hence the usefulness of a 2-phase process.

Here's a very silly and simple example: you have an engineering task to search for an ID in a tree data structure. You may know several standard methods off the top of your head or you may look up the best performing algorithm online. Either way, the solution is the expected one.

But let's abstract the problem: Why do you need to search for an ID in a tree data structure? Do you need a tree in order to keep track of the path and the IDs you pass? If not, then why not scrap the tree completely and go with a hash map of key/value pairs? That solution is both faster and easier to implement, but it only occurs if you think about the goal of the problem instead of the problem itself. You can abstract it even further and ask whether IDs are required at all, and if they aren't what other solutions could be used to accomplish the same goal. This is a very, very simple example but the idea can be applied to varying degrees in any problem, and kind of comes to the idea of finding first principles in physics.

As an engineer, I am aware that my mind often plots the shortest path from problem to solution, without ever stopping to question the problem definition. Sometimes reframing the problem is the best way to get to a more creative solution.

Often they ask for a solution to the 3rd step in their process where they got stuck and it turns out the underlying problem is easier/faster to solve

Think like a child, or like someone who would not have much idea about your subject. Allow yourself to make wild guesses, allow yourself to make shortcuts and unexplained decisions, in your mind, wander aimlessly. Move from one potential outrageous solution to another, without judging. Try to have fun with it. Don't note anything down. You think something is bad? Doesn't matter. You think something is good? Doesn't matter. Keep exploring.

After you are tired of this, stop. Try to write down what you remember. Then sleep on it, without looking at what you wrote before, write down what you remember again. Now you can look at what you wrote in both times.

Say you find a solution that is completely outrageous. How can you adjust a part of it to make it a bit less outrageous? Divide it to chunks. There will be parts you understand and parts you don't. There will be parts that are wrong and parts that are not. Divide and conquer.

Practical is good. What will actually kill your creativity is self-doubt. Stop doubting yourself and be happy with the way you are.

So, if you want to start to even out your levelheadedness then there are a lot of fun avenues to pursue. Creative writing, read some surrealist literature, talk with people who have experimented with psychedelics and witnessed the preposterous cognitive edifices of modern culture fall to shimmering disjoint pieces before their eyes, etc etc. Then work on reconciling the cognitive dissonances along the way.

The pragmatic mindset naturally simplifies, reduces, focuses, makes early judgement calls about what trains of thought are unlikely to yield results. The opposite of that is to be open and curious, to let thoughts expand and meander without any clear destination or utility and to see where those take you without assuming you know what is or isn't possible. That's where divergent imagination really gets going.

Each mindset serves a purpose; it's beneficial to learn how to shift between them as needed.

But from my experience with myself and friends, I think, sadly, this is typically not the case. People can act in either mode, but tend to have a proclivity, whether born or cultivated, for one over the other. You can do the other, but it will require extra effort for lesser results.

There are unicorns that can do both brilliantly. But much fewer and farther between than those who claim they can do both.

I've always been strongly on the engineery side myself, but have tended to fall in with artistic types socially. In my experience, artsies often fall short of being able to articulate ideas concretely, and thus can't easily interface with people that need a delineated plan. Which makes us engineeries tend to think they're soft brained. But in fact, they can often execute even highly complex systematic projects well. They just can't tell you how they did it.

If you're not getting enough ideas, think laterally, learn about other fields. See how chess players think, martial artists, football players, soldiers, food franchising, theology. Every field has its own 'common practice' which seems really obvious to the people in the field, but strange and new in a different field.

However, in the vein of this thread, I think that is only playing one note to learning CS. When I compare CS to martial arts (MA), I consider the traditional programming exercise akin to sparring. It is real world situated, but with the number of new variables it is difficult to successfully demonstrate or refine your understanding. MA, like CS, has a high attrition rate to start, and for reasons I consider isomorphic - a false sense of what CS/MA is and the demoralizing effect of "losing" constantly (sparring or debugging).

So, in MA, there are attempts to make sparring less intimidating. You can pair equal experienced partners or against an understanding senior, but you also focus on drills. If we consider sparring as a combination of problem solving and applying technique in the moment, then drilling removes the problem solving aspect to allow for technique refinement.

So, in combination with traditional programming exercises, my current work is looking at novel practice activities separate from tradition - for example, error finding, output prediction, and even typing exercises (which I have published on).

There are also sensory cues: You might choose a new location to do your gaming, play some appropriate mood music, etc. On one of my first ever "software guy does graphic design" projects, I asked the client what music they liked, and then listened to that for about 3 days straight. It really helped me get into the correct problem-solving lens where more appropriate colors, shapes, fonts, and textures were concerned.

On top of that you may find there are supplements that help you out. Personally if I'm running on 8+ hours of sleep and take about 100mg caffeine, I feel like I can solve just about any problem and new ideas are more likely to seem fun. ;-)

In the past I wrote some randomization software to help me solve problems. One of my favorites simply picks random words out of a dictionary. Each word is treated as a metaphor. I remember once a client said, "I thought I knew exactly what you'd say, but that is an idea that never would have occurred to me." This was my goal.

IMO there's a lot of space to grow in this area even if you feel like a potato where your imagination is concerned, and the process can be really enjoyable. Good luck to you.

Being practical means things that are doable. So it makes sense to limit whatever you do to that, otherwise you'll be in the domain of unrealistic / never gonna work.

The problem is, what is doable is not an absolute. What's practical is different for everyone. You'll define it based on what you experienced / heard about. Fact is you only hear what resonates with you, aka you're not even going to consider things that seem ridiculous, based on your experience.

There's two sides to every story and it seems you're missing one. In my view, being practical is a way of talking yourself out of ideas that you don't want to do.

Look at this way, if you stick to being practical you'll only every do things that have a close to 100% chance of working (in your view). This would only work if you knew everything. Absolute knowledge.

You have no idea what is possible, so do your best (keep trying) and don't overanalyze.

I've heard somewhere you can turn this mentality around by forcing yourself to come up with reasons this could work, rather than why it couldn't. Make a list.

You can learn a new mode of thought. You have to be open, and not dismiss things early, which means allowing yourself to follow ideas which have obvious flaws or which totally impractical. It helps to have confidence that you'll use your practical, analytical, and more closed mode of thought later. But first you've gotta jump out of the local maxima and try to find something new. Generate ideas as if you're high; highdeas are usually bad and that's ok, your sober self will be around to check it out later. Maybe the idea will be thrown away then, and maybe it won't.

There's a great John Cleese talk on open vs closed thinking that you could look up on YouTube. A good book on the subject is A Whack On The Side Of The Head.

Edit: That said, there's also something to working with another person, and each focusing on one of these roles more than the other. Like author and editor. Lots of things follow this pattern of splitting these roles up among people.

Since they haven't internalized the exact definitions of the words they're using or hearing, or even how the world works from a physical or logical perspective you literally can't replicate it with your adult brain. You know too much. It's kinda cool.

I constantly perceive that this is a major problem in modern UI designs, whereas in the past people thought really hard about usability, because they didn't grow up using computers, everything was new.

https://prog21.dadgum.com/39.html

Oh! Also try allowing your brain some breathing room. Walk away from the problem and do something else. Your brain will work on it in the background. The more you dwell on it in your conscious mind, the more you will focus your existing solutions down.

Edit: You must learn to trust your subconscious to help you with keeping an active eye on it. Just as you must trust employees to do their work without micromanaging them and breathing down their necks the whole time.

https://youtu.be/iHcxkmwBOJY

Then one person starts reading their list, and everybody checks their lists for that idea and cross it out. Then the next person goes, reading any remaining ideas and so on until all you have left are unique ideas (and it doesn't matter if all your ideas are crossed out---remember, no judgements yet). If you are alone, you can skip this step.

Then, and only then, do you go through the final list of ideas and discuss them. Here, you judge the ideas, reject some, combine others, mix, match, and puree until you get something that works.

Truly innovative solutions to problems are few and far between. Nearly everything is a combination or extension of solutions that have come before. Being good at coming up with ideas is mostly having a good database of knowledge to draw from, and taking the extension or combination process a couple steps too far before tossing out impractical ideas. This is why brainstorming can be effective, if you are working with someone where you feed off each other's ideas, the process can be a lot more natural rather than deliberate.

But when nothing comes to mind up front? If anything, I feel that my ability to sort the 'hey, that just might work!' wheat from the fanciful chaff is sharper than before.

That doesn't mean I can force it, and I think this point is important enough to emphasize- if I hit a block on an obvious next step, I stop and go for a walk. Listen to music, hang out with my dogs. Maybe play with a recipe in the kitchen, or create a new belt hell-world in Factorio. Let your 'default mode' network ruminate and digest. Nine times out of ten by the next day I've discovered a new approach to work with.

But a suggestion: try doing mundane actions like brushing teeth or writing on your lesser dominant side. Example: if you are right handed, try brushing your teeth with your left hand etc. try with various tasks. do this for maybe a week and let us know if you notice a difference re how you think.

I know of two school of thought regarding the origin of original ideas :

- Original ideas stem from ignorance of the existing idea and being an outsider in the domain. I don't think this is sustainable as, at some point, you have to become familiar with the domain to which you want to contribute.

- Original ideas comme from a deep knowlege of the domain and its history which lets you understand how new concepts you meet might fit in the overall puzzle.

Overall, if you have problems coming out with outrageous new ideas, I would recommend practice : whenever you enconter a problem, try to design a solution for it (it doesn't matter if you never implement it) and only then research what kind of solution is used to solve this problem. With time you should become familiar with your own creativ process.

In every application or machine I write code for, there is typically one, sometimes two, really interesting problems. But there are innumerable bits of boring business logic to implement along the way. You sometimes need to be willing to have a method copy-pasted with trivial changes in three places (which bothers me sometimes, thinking it would be better to have one method with a parameter to select between the variations) in order to have time to develop an extensible, adjustable, clever solution somewhere else.

Check out the writing of David Kelly and his brother (of IDEO) — they’ve probably done more than anyone else in the field to try and crack the formula of creative thinking.

If something seems to complex or hard, I figure their must be an easier way to solve it.

A lot of it is stepping back and questioning the original problem you are trying to solve.

"The man who has no tincture of philosophy goes through life imprisoned in the prejudices derived from common sense, from the habitual beliefs of his age or his nation, and from convictions which have grown up in his mind without the co-operation or consent of his deliberate reason. To such a man the world tends to become definite, finite, obvious; common objects rouse no questions, and unfamiliar possibilities are contemptuously rejected."

-- Bertrand Russell

You need to be able to turn this off during brainstorming. i.e. Suspend disbelief for a bit. Like watching a Marvel movie - live in the moment.

Broadly - quite the opposite.

The stuff my cash constrained cousins living in Namibia come up with blows my mind. Both practical and imaginative.

e.g. Building large RC planes from scratch (electronics excl). And here I thought my paper planes were grand.

e.g.

One of them got told there is no cash for uni, if he wants to go he needs to make it happen on his own. Guess who is now the proud owner of a bee keeping operation.

Anyway totally willing to be part of a imaginative/practical HNers coaching group!

A few months ago I started being extremely pragmatic in everyday life (even against my will which is usually not like that). Things like, say no if you want to say no, answer emails immediately with a sentence at most, find the fastest way to get something done etc...

Now I have a lot of time on my hands to do whatever I want, that includes spending the day studying or pondering things which I believe greatly increased the potential of my imagination.

Find things that you like and do them, weed out everything else. If you can't shake some of those away, find the most practical way to get rid of them.

Once I learned being practical first and creative only when needed, now I feel quite happy and productive. Seems like the satisfaction of my customers is much better than the tech itself.

Granted, the beginner is also going to go down a lot of dead-ends, but sometimes the expert won't get started on something which they have mistakenly ruled out as impossible.

The question is, what (mental) exercises could you do to get into the practicality equivalent of "beginner's mind"?

having endless debates on what's important for creativity : being flexible or being organized,

having read the book on this topic called "making ideas happen",

I got to understand what we all know already

To do the best of our work, it requires

Balance

- Balance between creativity and organization

- Balance between being practical and being a mad scientist

- Working in routine and breaking that routine

- Having a laser focus on problem and staying away from the problem

[1] https://en.m.wikipedia.org/wiki/Illywhacker

But the world values grounded workable solutions to real problems too. Being practical is good for these.

I find myself exasperated when, given a practical and an impractical solution, some people turn their nose up at the former as uninteresting.

Solving problems within actual constraints may be the greatest creativity of all.

Im too much of a dreamer/imaginary-driven, and i need these types to keep me focused.

One immediate example that springs to my mind is Albert Einstein. You could say that his ideas are outrageous that space and time are same and they are bent etc. Even people (like the Nazis) publicly called it outrageous and blasphemy. But if you think about his ideas, he clearly followed through with a rigourous mathematical proof which I think might be indicative of height of level headedness. In fact, if you skim through his process of explaining relativity to general public through his book [0], you cannot distinguish when he is being level headed and when he suddenly jumps into a bizarre thought experiment.

[0] - https://archive.org/details/cu31924011804774/page/n10

And whatever you do, don't share ideas with colleagues until you've sobered up.

To answer your question you should understand what creativity and practicality mean in your context. I would generally think that being practical and creative aren't at odds with each other.

When working through a problem for work I tend to err on the side of practicality. We could be coming up with a new way to do data transformations, building out distributed tracing tools, or trying a new way of scaffolding our projects. Each of those solve a real problem and have practical implications. They also require creative thinking to imagine and design new tools and workflows.

These projects are practical in the sense that they need to solve an actual problem (e.g. deliver a net benefit to customer UX or development velocity). Contrary to how they sound, they often aren't obvious projects to take on or straight-forward solutions. It usually required days or weeks of lateral thinking to cook up one of these. In this context, creative thinking complimented practical thinking and produced a better result than one of those modes taken in isolation.

Sometimes it's helpful to circle around the problem you think you want to solve to better understand what your actual problem is.

Be careful reading other's ideas since it will affect the way in which you see the problem. Your view of the problem may be skewed after reading other information. OTOH, it's common that you can piggyback on existing ideas (see taco bell/duct tape programming).

There are lots of good resources on this subject: John Cleese talk linked elsewhere - https://www.youtube.com/watch?v=y70nbDJI5Uk

Rich Hickey Hammock Driven Development is a more direct application to software development.- https://www.youtube.com/watch?v=f84n5oFoZBc

Another good Alan Kay talk from YC - https://www.youtube.com/watch?v=id1WShzzMCQ

A Mind for Numbers

How to Solve It

Psychology of Intelligence Analysis is interesting too but less applicable (eschewing the Einstellung effect) - https://www.cia.gov/library/center-for-the-study-of-intellig...