As an entrepreneur, my observation is that the vast majority of the experiments I run result in an outcome where the null hypothesis cannot be rejected. That is the case for most social science experiments (which is what marketing experiments really are.) But since I don't have to publish anything, I don't p-hack my results.
If you think that scientific rigour will help you avoid the need for good judgement, you're in for a great deal of distress.
Edited for typos.
> If you think that scientific rigour will help you avoid the need for good judgement, you're in for a great deal of distress.
Broadly, assuming more data and tighter calculations == more certainty is a folly endemic to engineering-focused organizations and industries. In reality, you're increasing precision without increasing accuracy: a dangerously misleading state. It's understandable: organizations obviously need to play to their strengths, but there's real danger in succumbing to the "when you're a hammer, everything looks like a nail" mindset. Firstly, It's much easier for non-engineers to parse the difficulty of engineering problems than the other way around. Outsiders easily see that they don't have the requisite chops to do engineering work. Secondly, cutting through ambiguity and unpredictability to reveal factors you can control is critical to engineering, while non-engineering jobs like management, design, and community outreach are hard because you must confront those things. Engineering types often reason about non-engineering problems either like they have the same level of predictability, or simply disregard the ambiguous and unpredictable factors because they don't fit into an equation.
It's easy to see why that micromanaging non-technical manager screwed up by insisted on using some technology or approach they read a snappy article about; an engineer's authority is cut-and-dried in that situation. It's harder to see why purely formulaic approaches often fail dealing with people, nature, markets, etc. Most things in the world are far more complex, temperamental, and less predictable than cache invalidation.
> After a candidate's defeat in an election, you will be supplied with the "cause" of the voters' disgruntlement. Any conceivable cause can do. The media, however, go to great lengths to make the process "thorough" with their armies of fact-checkers. It is as if they wanted to be wrong with infinite precision (instead of accepting being approximately right, like a fable writer).
"Firstly, It's much easier for non-engineers to parse the difficulty of engineering problems than the other way around. Outsiders easily see that they don't have the requisite chops to do engineering work"
It's probably worse in business types because they haven't gotten the exposure to science enough to realize that a lot of things remain tangible despite being un-quantifiable.
The same is true in science too; which is why being an experimentalist requires lots of skill & judgement to tease apart the signal we want to understand.
Scientific rigor is inseparable from good judgement.
I agree and I don't think it is specific to the world of entrepreneurs and/or social science experiments. If your target variable depends on a large number of correlated variables you are very unlikely to formulate the correct hypothesis by accident. This is why you need intuition or good judgement in science, just as much as in every thing else.
And if you don't reject the null hypothesis in the vast majority of experiments, then a large portion of those where you do were probably also just a statistical anomaly.
> that scientific rigour will help you avoid the need for good judgement
This is what all big company PMs and directors of engineering believe. But their judgement (good or bad), their metrics (scientific or pseudoscientific), their career trajectory (up or out), whether these are known or unknown in the first place, and their actual material success: who knows how correlated it all is. They don’t have the words to express this stuff, so they wouldn’t be able to see the difference between science and pseudoscience for example, and you'd have a hard time communicating this thing about science to these people.
So while you are right, it’s not saying much that the real problem is communications, and that is the mainstream opinion of people who study microeconomics / the structure of firms. Put differently, people have built cathedrals of bullshit in their minds in service of the status quo where they “test” “everything” in lieu of having falsifiable, forward-looking opinions (aka judgement). You can’t just go, Martin Luther all of corporate white collar bureaucracy.
If you think that scientific rigour will help you avoid the need for good judgement, you're in for a great deal of distress. Edited for typos.