Those aren't counterfactuals, they're negative statements.
"Why are the lights off?"
"Because I didn't turn them on."
That's not a counterfactual, it's a fact. Just like the admins not configuring file purging is a (presumed) fact in the scenario under discussion in the article. Negative statements are not counterfactuals. Now, they may not be helpful for a 5 Whys analysis, but that's a separate thing.
Counterfactual: If the admins had turned on file purging, then the volume would not have been full.
Why? Because they didn't turn it on so this is a counterfactual in that it is dealing with facts not in existence in the reality under examination and reaching a conclusion of how things would have been different with those different facts. But nearly all the causes in a 5 Whys can be seen as counterfactuals if phrased correctly:
"Builds did not complete." Why? "Kubernetes could not start the pod, and the operation timed out after 1 hour."
As a counterfactual: If Kurbernetes could have started the pod, then the operation would not have timed out after 1 hour and the builds would have completed.
"I didn't turn them on" is a fact. It's the "because" that's not a fact. The whole point of the article is that you can't get causality from things that didn't happen. There are seven billion other people in the world who also didn't turn on the lights. What makes your inaction special?
In order to get the "because" we have to imagine another state of the world where you DID turn on the lights, and further speculate that this would have been effective to illuminate the room. That's counterfactual, because the fact is that it's dark in here.
But the because is not about imagining another state of the world, it's about using logic.
If me turning on the light is a necessary condition for the light being on, then it implies that if I did not turn on the light the light is off. There is no speculation. This is also what distinguishes this one "counterfactual" from all the others.
The article seems to completely ignore that we can readily argue about negative statements. I highly doubt that airplane crash investigators would avoid counterfactuals.
There are no necessary conditions in the physical world. If you hadn't turned on the light, perhaps someone else would have. Or perhaps a bird or a meteorite could have flown through the window and struck the switch in just such a way as to turn it on. Or perhaps a wire could have come loose inside the switch and fallen in just such a way as to close the circuit.
Sure there are necessary conditions in the real world. It is a necessary condition for the apple to hang on the tree, for it to fall from the tree onto Newton's head.
By arguing there are no necessary conditions you are essentially saying we can't use mathematics to describe nature and there are no laws of physics.
For the apple to fall on Newton's head? It could have been dropped by a passing swallow (or possibly two, held on a line between the dorsal guiding feathers).
For it to fall from the tree? Well, that's a linguistic trick; for it to fall from the tree it must have been hanging on the tree, so that necessary condition is actually a tautology.
Physics describes a world with statistical regularities, not absolute laws. Even a clockwork Newtonian universe is not time-reversible; it has multiple pasts that can lead to the observed present.
It's not a tautology to say for an apple to fall of a tree it has to be on a tree, it's not a tautology for a plane to crash to be flying first or for a car to stop it needs to be in motion first. They are all necessary conditions, in fact this is the definition of a necessary condition.
Something to be a necessary condition has nothing to do with time-reversibility, it simply states a logical relationship
Let me give you another example: To accelerate a body with mass m it is necessary to exert a force onto it. All mathematical descriptions In fact equivalency, requires conditions to be necessary and sufficient.
Your argument that physics describes a world with statistical regularities is a maybe a nice philosophy, however most physicists (and regular people) do not believe we live in a simulation, but physical laws describe the real world.
Yes; a simple example is Norton's dome. It is possible to construct a frictionless curve such that a particle that is pushed up the curve with correct initial velocity will reach the top and halt in finite time. Observing a particle sitting at the top of the dome, there is no way to tell when it reached that point or from which direction.
I think it is hard to answer this without reducing the real world into something it may not be. What constitutes an event in the real world? Is the universe discrete or continuous? Do we really use mathematics to describe nature, or do we use it to model and approximate some properties of it?
I think the article doesn't say that the counterfactuals are not useful, it explicitly mentions they can be used as recommendations, as in "do not forget to monitor your disk space".
What they say is that counterfactuals stop the search for the root cause. Not monitoring disk space isn't a root cause of the filled disk. The root cause might be "application has suddenly written too many logs". This gives another set of counterfactuals, like "if your application was configured properly, it would write less logs". But it also gives an option of continuing the search for the root cause, which might be, for example, "there is a bug in the application which triggered the log writes".
I think the point of methods like 5 whys is not to stop at the first set of counterfactuals and deepen the search.
Let's take a different example, the plane crashed because the landing gear did not retract. This is a counterfactual according to the article, do you really believe it stops us for looking at the root cause for why the landing gear didn't retract? In fact, I don't think we would be able to even investigate the root cause if we hadn't first identified the cause of the landing gear not retracting.
I think your example is not a counterfactual, because in your world, landing gear actually didn't retract. Remember, whether something is counterfactual or not depends on what the facts are.
Had the landing gear been retracted in your hypothetical world, then it would be a counterfactual, but it probably wouldn't arise because it's not something out of ordinary - the landing gear retracting is a usual procedure. I think humans are more prone to make the mistake that is described in the article (stopping to search for the root cause) when they see a counterfactual conditional which premise is, in the factual, a procedural error (for example, "operator failed to monitor the disk space"). We see an error and mistake it for the root cause.
If me turning on the light is a necessary condition for the light being on, then it implies that if I did not turn on the light the light is off.
To be pedantic, it’s not quite that either. The logic theorem states if A => B then !B => !A. The => means that A is sufficient for B, because “false => true” evaluates to “true” but “true => false” evaluates to “false”. If it were “necessary”, those statements would evaluate the opposite way.
This means that if me turning on the lights is sufficient for the light being on, then the light being off is sufficient to deduce that I did not turn it on.
You are simply stating that me turning on the light is not a necessary condition, but a sufficient condition (which is a reasonable argument).
My argument was that if the statement "if the light is on it was turned on by me" is true (i.e. the me turning on is a necessary condition), then if I did not turn on the light the light is off [edited after mistyping before].
I think this works better with other examples, because as you state, we typically think more of the turning on as a sufficient condition.
Yes, it wasn’t my intent to contradict your overall point, just to correct your logic.
My argument was that if the statement "if the light is on it was turned on by me" is true (i.e. the me turning on is a necessary condition) then it follows that if the light is off, I have not turned it on.
I disagree with you here. The counterfactual, or at least the logic theorem, for the statement [light is on] => [I turned the light on] (me turning it on is necessary for it to be on) is [I did not turn the light on] => [the light is not on]
That’s sound, though it’s not a particularly insightful leap of logic.
If you translate this to the author’s point, they’re basically saying that !A => !B doesn’t tell you that A => B. And that’s correct as far as Boolean logic goes. I am unclear on whether we disagree on that point.
Sorry I edited my post just after posting it, because I realised I wrote it the wrong way around (too many nots and light switches ;). You managed to reply just in between.
>I disagree with you here. The counterfactual, or at least the logic theorem, for the statement [light is on] => [I turned the light on] (me turning it on is necessary for it to be on) is [I did not turn the light on] => [the light is not on]
Yes you're correct and that's what I corrected my statement to where you managed your reply in between.
However my original statement was:
>If me turning on the light is a necessary condition for the light being on, then it implies that if I did not turn on the light the light is off.
so A=>B therefore !B=>!A. Where A == "light is on", and B == "me turning light on". I agree that it's not particularly insightful.
My point was regarding the statement "The whole point of the article is that you can't get causality from things that didn't happen" from the OP I replied to. That's just not true, we can make statements about causality for things that didn't happen, because we can deduce them from logical conditions.
>If you translate this to the author’s point, they’re basically saying that !A => !B doesn’t tell you that A => B. And that’s correct as far as Boolean logic goes. I am unclear on whether we disagree on that point.
Although we agree in principle I disagree what you say is the authors point.
From the article:
>They [counterfactuals] express wishful thinking about an alternate history where the bad event didn’t happen. Because they represent “events that didn’t occur” they cannot have caused anything.
This is a statement I disagree with. To me that sounds that the author essentially says we can not make statements of the type of !A => B (or !B). To me that is wrong and also not a counterfactual. Counterfactuals, are statements about a hypothetical event, i.e. "If I had not turned the light on it would be off now". To me though the author seems to extend the definition of counterfactuals to mean "events that did not happen", maybe he means the correct thing, but both his words and examples do not reflect the correct meaning IMO
Are you saying that "If I had turned on the lights, they would be on" wouldn't be a true statement?
It seems like it would be strange to reject as an answer to "why do blind people tend spend less on lighting?" with "Because they don't tend to use the lighting, because the lighting isn't useful to them, so they don't turn the lights on as much (unless someone else who isn't blind is there).".
How else would you answer that question? (modulo little wording changes and such)
It's tautologically true, in that you used the word "on" in both clauses, and "if on, then on" is always true.
But no, in a meaningful sense it's not a true statement about the world. It's a good guess, and for a statement that simple, there might be no need to go further. For a statement about why a volume is full with too many files, the point is that a nice-sounding "if only" isn't something you can test or be certain of.
Your answer about blind people is a very good guess as well, and the same one I would make, but it's not data.
I actually meant to present that as: Because I didn't flip the switch.
A counterfactual requires posing a situation with changed facts. I stand by my statement, saying "X because not Y" is not a counterfactual on its own. "not Y" is still a (potentially) factual statement. And guesses still aren't counterfactuals. "Maybe the lights aren't on because we didn't flip the switch?" That's a hypothesis, and a testable one. Still not a counterfactual. If we know the flip hasn't been switched, then it's just a fact even if the statement is in the form of not X.
The counterfactual is "if I had flipped the switch, the lights would be on." That is a situation with changed facts, and therefore counterfactual. The tricky bit is the causality. We have to construct an alternate history, with a counterfactual cause and a counterfactual result. And we can't actually know that. Maybe the bulb is burnt out. Maybe there's a power outage just now. Maybe we've blown a fuse. The causality is just conjecture.
Now consider the opposite. The lights are off because the bulb burnt out. That's a lot less ambiguous. If you want to doubt the causality you've got to get all mystical.
"The term "Counterfactual" is defined by the Merriam-Webster Dictionary as contrary to the facts.[2] A counterfactual thought occurs when a person modifies a factual prior event and then assesses the consequences of that change.[3] A person may imagine how an outcome could have turned out differently, if the antecedents that led to that event were different. For example, a person may reflect upon how a car accident could have turned out by imagining how some of the factors could have been different, for example, If only I hadn't been speeding...."
The forms the article mentions are like those "if" statements.
>That's not a counterfactual, it's a fact.
That's because there's a one-way casual relationship there. Not the case with TFA examples.
The main idea: Let Y be the set {y, not y} and X be the set {x, not x}. y denotes timing out, and x denotes “Kubernetes not starting the pod”. We have some distribution P(Y=y|X=x) (in this case, the probability of “timing out” (y) given “Kubernetes not starting the pod” x. The counter factual distribution is NOT the same as P(Y= not y | X= not x) (the “probability of not timing out given Kubernetes starting the pod”). The counterfactual is P(Y’=not y| X=x, Y=y, X’=not x), or “the probability of not timing out given that it did originally time out when Kubernetes did not start the pod, and given that this time it did start the pod”.
This aligns roughly with what you are saying above.
I will say that while I think this article doesn’t give a good treatment of counterfactuals - the “I can offer infinite counterfactuals so counterfactuals aren’t useful” argument is a kind of intellectual nihilism I don’t really buy - I think at its core this article is really just arguing that we should pose forward facing solutions when we encounter bugs, rather than explanations. That proposition seems fine to me.
I think those counterfactuals can be the most useful explanation, while not being the direct cause.
"why did the disk fill up" would be most directly explained by the most recent write. That's not useful though. It doesn't really matter what the last write was.
I agree with you. This author not only doesn't know the basics but is making an absolute fool of themselves by being ignorant of the robust and very popular (within analytic philosophy) research on counterfatual theories of causation. [1]
Based on the (apparently accidental) use of terms of art in the field, I clicked on this article genuinely expecting to see a refutation of Pearl or something.
"I'm tired because - I stayed up all night playing games", or, "I'm tired - because my child screamed all night long," are more along the lines of what the post suggests are useful, and I can see how they're fundamentally more useful to getting at the root cause than the counterfactual, "I'm tired because I didn't sleep."
But this example is very simple and makes the argument less important I'd say.
All three statements using "because" are counterfactual. If you said "I'm tired and I stayed up all night playing games." you're only making a factual claim. But using "X because Y" implies that Y matters for X, i.e. in the counterfactual situation where Y is not true, X wouldn't be true either.
The author of TFA seems to suggest that explanations where Y is "not Z" are inherently less useful (I don't think so) and also calls them "counterfactual" even when "not Z" is a fact, which is confusing.
'All three statements using "because" are counterfactual.'
No - the definition the article refers to is this: adjective - relating to or expressing what has not happened or is not the case.
I used "because" to illustrate the "looking for root cause" use case, like what the article was referring to. Regardless, that doesn't make it counterfactual per the definition the author is using.
To me the part that makes the counterfactual less useful when determining root causes is the human element the author mentions - looking for root cause tends to stop once they're brought up (they're the end of the causal chain in discussion), and blame starts to get assigned. I've seen that happen, and it hasn't helped resolve the issue, so avoiding that seems useful. I don't see a problem with using a counterfactual statement if it really is the root cause and I'm blaming myself for something.
> By this reasoning, "I'm tired because I didn't sleep" is fundamentally different from "I'm tired because I exercised a lot".
Well, they are different because they propose different causes. Either of these statements (and the so-called counterfactuals in the article) might be true, depending on the actual cause.
The distinction that matters here is that statements of causes are not solutions, and solutions are elicited by different questions than are causes ("what can we do about it?", rather than "why did it happen?") It does not really have anything to do with counterfactuals vs. causes, and the author's actual point seems to be about how to present solutions.
Funny enough, reductions of the causal relation to a counterfactual one is a common theme in philosophy (and adjacently in statistical sciences - see Pearl). Most philosophers would contend that a true counterfactual can be sufficient condition for causation, but not a necessary one. From that seed there are quite a few different accounts (David Lewis' being the most famous) that attempt to reduce cauation to counterfactual relations + something else:
Its still a somewhat live debate in contemporary philosophy.
The medicine saved the patient's life is true if the patient would have died had the medicine not been administered. How do we determine the truth of such a counterfactual?
In practice we do large scale randomised trails to attempt to simulate a counterfactual to test the causal powers of the the medince - but its probably beyond our powers to know for certain that the counterfactual is true in the singular case.
Philosophers are interested in what makes a counterfactual truth or false, and whether understanding a counterfactual presupposes causation or whether a causation can be understood in terms of counterfactuals. Nobody doubts that counterfactuals and causation are releated, and few doubt that counterfactuals can be reduced to their logical or epistemic counsins.
The counterfactual has never seemed epistemologically satisfying to me, because of the way the influence spreads. If I had in fact flicked the light switch, what is taking up the empty space where my body would have been? What happens at the power plant which is now generating a bit more current than it was before?
Implicitly there's this idea that every counterfactual can be contained and localized, but that runs counter to my understanding. Change one thing and you change the entire universe. That's true even classically -- even before you bring in quantum physics and its problems with counterfactuals.
Perhaps counterfactuals are a good model of the psychology of causation. And in fact, I suspect that psychology is as deep as causation runs; it doesn't in fact have a direct physical or epistemic analogue. All of which is fine and important, but I feel like it diminishes the value of the effort put into it. It's not studying VLTIMATE TRVTH, just a quirk of how brains think about stuff. And that would mean you need to use a very different toolset to work on it.
This post would have been stronger if the author had gone back to their original 5 whys and continued it without counterfactuals.
- Build didn't happen, why?
- Pod didn't start, why?
- Disk was full, why?
At this point if counterfactuals are off the table, the answers become tautologies. Although if everyone is in agreement that counterfactuals are not helpful, this could become a powerful heuristic in determining that you've hit root cause and can shift from searching for root cause to designing mitigations for the root cause.
I guess that's the point of the post, but it seemed weird to start a scenario saying "it goes off the rails here, because of XYZ" and not come back and give the better version of that scenario.
Definition: "A counterfactual is a statement about how the world might be different now if something had happened differently in the past. It’s a kind of “alternate history” idea."
"Here’s the rub: a counterfactual cannot be a cause. By definition the counterfactual did not happen, therefore it cannot have caused anything. Only events that actually occur can be causes of other events. Causality should be stated in a form “Because X then Y”. The statement “If not X then not Y” is not an explanation, it is a kind of wishful thinking about how the past might have unfolded differently."
"When performing Five Whys it is important to avoid this counterfactual leap. Stick to the events that actually occurred. [..]"
"Try to reformulate the counterfactual as a statement about future prevention:"
1. If we configure file purging, then this won’t happen again
2. If we monitor for “volume full” conditions, then this won’t happen again
3. If we clean up files from old builds, then this won’t happen again.
These are useful statements. When formulated this way, they’re clearly talking about the future and not hypothesizing an alternate history. "
I mostly agree with this author but I think counterfactual isn't the most useful word in the article.
For example "The admin did not configure file purging." isn't a counterfactual unless admins were in fact purging files.
A better phrase might be subverted expectations, along with the admonition to make expectations more explicit with the author's if-then examples. Inaccurate expectations are just as important to analyze and fix as hardware and software bugs.
Counterfactuals are actually very powerful and useful because they do let us analyze the past instead of just the future. I have found that chasing counterfactuals usually bottoms out in known failure modes in practice. For example tracing back that lack of disk space in build nodes would probably arrive at either "insufficient design" or "insufficient review" or even "insufficient understanding", all of which are pretty well-known and common engineering failure modes and we already know the solutions in the future; spend more time on design, review with more people, ensure education is happening. It's actually quite useful to do this because as a trend over many postmortems "insufficient design" has very different organizational solutions than "insufficient understanding" and it's a meta-level counterfactual of "if we were better at (design|understanding) what problems would we avoid?"
Bro lost me at "Only a finite number of events actually happened..."
Actually, he lost me earlier, cos "The admin didn't configure file purging" is something that did happen: the admin failed to take an action. The event happened even if the action didn't; the article confuses the two
It doesn't really matter tho since this article is really just a piece of advice on how to handle incident reports in a tactful and constructive manner, not a serious argument about causality
This seems to rely on a distinction between positive and negative statements. I'm not sure whether that that distinction works everywhere? (e.g. is "x is less than or equal to y" positive or negative? what about "x is less than y"? The negation of the first of these (under the assumption that x and y are comparable) is "y is greater than x". So, it seems that the two have to have opposite uh, .... I'm going to say "polarities"? But we can also say that "x is less than or equal to y" is the same as "either x is less than y, or x is equal to y", and, err if the conjunction or disjunction of two positive statements is positive, and similarly of two negative statements is a negative statement, ...? I feel like maybe something goes wrong here, but I'm not sure.)
___
If there is a domino, and I push it, and it falls over, I think most will agree that "It fell over because I pushed it.".
If I don't push it, this article seems to say that "It didn't fall over because I didn't push it." is not a causal explanation.
And, maybe that's right, but I don't think the "because 'if I had pushed it, it would have fallen down' is a counterfactual" is convincing reason for that.
Because in the case where "I don't not push it" and it "doesn't stay up", shouldn't "It didn't stay up because I didn't not push it" be a valid causal explanation, being equivalent to "It fell down because I pushed it",
yet, in this case, "if I hadn't pushed it, it wouldn't have fallen down" would be a true counterfactual?
-- oh, this article is about advice for language use in interpersonal stuff (especially in a work environment), and maybe practical problem solving in general, not studying metaphyics-or-whatever of causation, ok.
I found this insightful. However, it does seem a lot of the discussion is around the concept of causation by absence. I don't think it directly impacts the point but there is a good paper by Helen BeeBee [1] on the topic for those interested.
She discusses some views on how to reason about common sense statements like "Smith's failure to water her office plants caused their death". The salient views covered are:
1. causation by absence is impossible (making the plants statement false)
2. causation by absence is possible and attributing an absence to a cause is sound when the absence is "abnormal". This could be because Smith normally waters the plants, it was Smith's job to water the plants, or Smith made a promise to water the plants.
3. causation by absence is not possible, but it is valid to offer "causal explanations". This is the suggested approach in the paper. A causal explanation provides information about the causal history of an event, but doesn't need to specify a specific node in the causal graph. C causes E is a causal statement. E because C is a "causal explanation". E because C doesn't necessarily imply C causes E. So adopting this approach, The plants died because Smith failed to water them is a true statement that provides some additional explanatory information, Smith failing to water the plants caused their death is not.
I think these distinctions may have some minor implications on how you perceive blame.
1. The point of counterfactuals is not to "prove" causality; it's to help you detect possible causal routes, in the presence of given events under examination. E.g. if you know events X, Y and Z co-occurred, and you want to know if X or Y could have possibly caused Z to happen, the counterfactual formulation can answer that question. If not X implies not Z, whereas Y does not alter Z, then you have eliminated Y as a causal agent, and have identified X as a possible causal agent (in fact the only possible causal agent, if Z could not be influenced by another external, unidentified agent, or by itself).
Furthermore, in the presence of a number of counterfactual statements, you can allocate responsibility and blame, linked to expectations of events. So aliens is a silly example, unless for whatever reason the author has a very high prior expectation for their existence, which exceeds that of file purging errors.
Many times I dont have visibility into the entire system so the best I can give the client a good set of counterfactuals on the bad policy they used to get to that place.
Is it fair to say that the counterfactual is hypothesizing the B in an A/B test?
I.e. I administer a pill with an active drug to a treatment cohort. What would have happened had I administered no drug (sugar pill) to the same treatment cohort.
The lawmakers should read this article, take it to their hearts and rewrite criminal codes all over the world to not punish people for the lack of action.
I was trying to find out what rubs me the wrong way in this article, and I think it is this:
Counterfactuals require a causal model. More so, they belong to at least one causal model. This can be a mental model, a formal one, a precise one or a diffuse one. Nevertheless, counterfactuals "come" with an implied model. People may disagree about which model they talk about, and which model is correct. Such a model may or may not be "precise" enough to identify useful parameters, which is the real rub here.
For example, the three "Counterfactuals" the author presents presume a model where purging the cache (etc.) directly causes the problem to go away. Each of those statements might come from the same or from a different causal model.
The problem, which I feel make this approach not useful, is that the author has no model in mind. In other words, they switch between different models. One counterfactual is, for instance, "If we don't live on earth, then we don't care about the cluster". Let's ignore that the outcome variable is different. The model with "We are not even alive" as a variable is not a reasonable one. First, it is empirically wrong. Second, it does not "identify" parameters (relationships) of interest. These two points are what we strive for when using a model (which we do when we construct counterfactual, even if it is a verbal model!)
The author posits that one can add unlimited counterfactuals. What the author means is that one can add unlimited variables and relationships (however diffuse) to the model. This is true.
However, we still need to judge whether the model is good, whether it is helpful.
In drawing such analogies to statistics, one should not forget the other concepts.
From a statistical standpoint, the author begins with a model that does not identify causal parameters associated with the failure to start the cluster. They then add "counterfactuals", that is, they expand their mental model to new causes. They find that any of these variables may cause a counterfactual where the cluster starts.
In my view, the author is only half right. Counterfactuals, as presented here, imply a causal model (perhaps not a precise one). Counterfactuals can be useful, if the underlying model is useful. Whether this is the case, depends largely on factors the author does not discuss in the article.
Edit: It may also be helpful to distinguish a counterfactual from "observations with exogenous variation". For example, if we purged the cache some time ago purely by accident, and the cluster did start, we might take this as a hint that purging the cache allows the cluster to run. This is not a counterfactual. It can be used to identify causal chains. If we were sure that nothing else was different (in a certain statistical sense), then we could "identify" a causal path from cache purging to staring the cluster. However, to determine that, we again need a model. In fact, we need a model and assumptions about the observation (what is called data generating process) jointly - because really, they are the same thing. For example, if the cache was purged because an engineer discovered an error somewhere, and then did some other action Y - perhaps it was not the purging of the cache that allowed the cluster to run. It may have been action Y. Here, purging the cache was not "exogenous" enough for us to infer the correct reason.
Identification of a model/DPG is a deep concept, but a crucial one to finding cause and effect.
The author doesn't know what causation is. Heck not even correlation establishes causation so OF course a counterfactual can't establish causation.
Not only are counterfactuals infinite in number but ALL hypothesis are infinite in number. You can literally say ANYTHING about why something didn't build. I can say someone didn't purge some files or I can say someone purged files or I can say aliens destroyed the server with a laser. There are an infinite number of statements that can be made here and whether it's a "counterfactual" is irrelevant.
What the author doesn't know is that NOTHING can establish causation except for a very specific type of experiment.
Basically there is nothing that can establish causality UNLESS you have control over the causal switch. You flick the switch randomly and if the randomness of the switch flicking correlates with the consequence you have then have established causation. BUT you have only established it to a certain correlative degree. It is fundamentally impossible in this reality to establish causation to be 100%. True causation can never be fully verified and at best the experiment I described is the best fuzzy verification. If you truly study how science works you will find that this flaw extends to all of science and therefore reality. In reality as we know it, nothing can be proven. Proof is fundamentally impossible. If you understand this concept then you will truly understand what science is.
Anyway, Going to the authors Kubernetes example, his "counterfactuals" are basically null hypothesis and can be helpful in establishing causation but by themselves are pointless.
It's simple... If you hypothesize that something didn't build because the admins did not "configure file purging" then to establish causation is SIMPLE. Configure the file purging and see if your system builds. Then remove the configuration and see if the system stops building. Do this enough times and you have established causation or zero causation to a certain degree.
That's it. This is science and probability and statistics. So crazy to see someone mention causality without even knowing what it really is.
You hint at it, but causation is a fuzzy concept even with a RCT.
Causation is "established" in the usual sense by having a causal model of a data generating process. Such model may or may not identify causal parameters given the data. That is, it is not the "randomized control trial" itself that establishes causation, rather it is a more general concept of statistical identification.
Case in point, there are instances where a RCT fails. For example, even if you can assign treatments, you might find that treatment and control groups influence each other or whatever else can go wrong in a RCT.
By contrast, there are instances where we do not control the causal switch, but we can observe "natural experiments" nonetheless. Usually, causation is fuzzy, since there are at least some assumptions which need to be made for us to believe the analysis to be causal. However, really, the same is true for an RCT.
For example, if you observe an engineer sometimes purging the cache, and sometimes not purging the cache, you do not actually control the assignment of treatment. However, it is easy to think of a mental model where observations of this engineer, and the state of the cluster, gives you high confidence to proclaim: "Purging the cache makes the cluster start!"
So, this is just to expand on what you wrote: Causation is not a precise concept, because it depends on our model of the DGP and the data we actually have. Identification of causal parameters depend on these factors, and thereby on assumptions - this is true for RCTs where I control the treatment, just as much for observational studies where I do not. Even if I assign treatments, I have to make the assumption that my assignment is truly random, and that there is no non-random factors influencing my observations (etc).
In quite a broad sense, causation as a scientific concept depends on assumptions I need to make.
If I get to control the treatment variable (or variable of interest), it is typically easier to create "exogenous" variation that identifies causal parameters... in English: Allows me to establish causation.
However, the whole concept does not depend on controlling or not controlling treatment. RCTs can fail, or there are instances in which I can establish and measure causation even without controlling the treatment.
It may be that I'm coming from a more philosophical perspective. What is randomness?
If randomness involves free will then only an entity with free will can create that random variable aka only another person or yourself can create randomness.
Pedant-ism aside, can you think of a "truly random" data generative process that doesn't involve someone with their hand on the switch?
I guess this is the assumption I am making in my explanation: Only people with this magical thing called "free will" can generate random data, nothing else can be trusted to be truly random so you or another person has to have their hand on the switch in order to establish causality.
Note that even your example stayed within the limits of this assumption. You were possibly unable to choose another example that didn't involve someone with free will. Either I have my hand on a switch, or I'm observing another engineer who has his hand on the switch.
Typically to illustrate a point one would usually pick an outlandish example that strays very far from a human having his hand on a switch and show that the example is STILL true. But you didn't do this and that may mean something.
I cannot really engage with your question. First, because I am not an expert in philosophy (a Philistine, if you will). Second, because it is not axial in the identification of causality, such as we normally think about it.
You see, in the pragmatist view of a statistician (and in the math that goes with it), randomness - for example in the form of statistical independence - is not required to identify causality (or causal parameters of a model).
Instead, we need the "switch" variable to be uncorrelated with things we do not observe, or can not adequately model. This requirement holds for a random experiment as much as for an observational study. However, you will quickly convince yourself that this is an assumption. It depends on the data and how we got it, it depends on what we assume about the factors at play, and it depends also on the model and how we take into account factors that influence the outcome.
Specifically, the identifying assumptions for a causal model are untestable. Let me emphasize this again: Every causal analysis is fuzzy, because they are only causal based on the assumptions.
As such, the question of true randomness is somewhat ancillary: Statisticians know that the idea of causality is by default not a philosophically "true" concept. This, of course, leads to a pragmatist view.
One last point: Between free will and flipping a switch, and a causal analysis, stand many assumptions about the data, the measurement, and so forth. In that sense, statisticians probably do not feel that a random flip switching is less assumptive than any other causal analysis.
Related to this, let me offer you the perspective of social scientists. They feel that human behavior IS influenced by other factors, just as a machine or physical system would be (broadly speaking). That is, they would reject the notion that free will is indeed (or at least always) random.
In such studies, for example, choices from individuals would also be subject to intense scrutiny with respect to unobserved influences or other factors that might endanger the exogeneity of the observed choices.
Causal identification has been likened to a Zoo, with many kinds of animals in them. I am sure the philosophical idea of causation is part of the Zoo. On the other hand, statisticians speaking of causal analysis find themselves unable to "prove" any sort of causality, even weak forms as indicated above, without underlying assumptions. Case in point, a randomized controlled trial is chock-full of identification and exclusion assumptions. They are just standardized and accepted.
Sorry again that I have to rely on the statistical view - it is the only one I know a little bit about.
>I cannot really engage with your question. First, because I am not an expert in philosophy
I feel the entire field of philosophy is mostly useless and full of categorization errors. I don't buy into it, I just use the word for speculative stuff. A lot of the things that the actual field of philosophy chooses to speculate on (Animism for example) is just random garbage. They also pair religion and logic together as two peered pillars of study, as if religion is just as fundamental as logic. I have little respect for the entire field from a formal perspective. In my reply, I was using the word informally.
>You see, in the pragmatist view of a statistician (and in the math that goes with it), randomness - for example in the form of statistical independence
No we're talking about randomness as a fundamental phenomena in a procedural and physical universe. Meaning that what is a definition of a pure stateless function that takes no input and will always output a completely random number? The probabilistic definition of a random variable does not actually produce such a function and sort of avoids the concept of randomness all together.
In physics, quantum physics specifically, such a function is not defined but is taken as an axiom. The actual location of a atom after decoherence is assumed to be axiomatically random.
>Instead, we need the "switch" variable to be uncorrelated with things we do not observe, or can not adequately model.
Arguably everything in the universe is correlated and shares a causal connection. A butterfly flapping it's wings in China causes a hurricane on the other side of the world. On a grander scale: A perturbation of an atom in one corner of the universe eventually changes the parameters of every atom in the universe.
According to certain theories in physics information can only travel as fast as the speed of light. Therefore according to this logic all things can be causally connected but only up to a point. If you imagine multitudes of infinite spheres continuously growing out of an atom at the speed of light, then everything outside of a sphere that started growing at t0 is guaranteed to not have a causal connection to any events that happened to the atom at t0 or after.
However in math we create our own playground devoid of all the messiness of the universe and it's atoms and limits to causal transmission across large distances. Within this world there seems to me, to be an issue with your notion of a switch to be "uncorrelated."
For something to share a causal connection with another thing, both have to be correlated. You cannot establish causality if the switch is uncorrelated with the observation. The result will always be a failure in establishing causation if you are selecting this type of switch.
There is a paradoxical problem with the definition here. I get your intuition and you understand mine as well but this definition seems to be incomplete or not an accurate illustration of both of our intuitions of this concept.
Using "randomness" as a concept and leaving "correlation" out of the definition seems to better fit our intuition of what's going on when *I* or another human flips the switch. After all, you cannot have a causal relationship without a correlative one.
>Statisticians know that the idea of causality is by default not a philosophically "true" concept.
I would say the just "don't go there." They stop at the definition of a random variable and don't dig any deeper. https://en.wikipedia.org/wiki/Random_variable <--- on wikipedia a random variable is defined informally in terms of "random phenomena." Additionally the formal definition is defined in terms of "possible outcomes" so they're really side stepping the problem here. But that's ok, it's not really needed to play the mathematical game of probability.
>Related to this, let me offer you the perspective of social scientists.
A social scientists opinion is not any more valid than an average persons opinion. A social scientists views human behavior as a black box. They don't just put one human brain in a black box, they actually put a population of human brains in a black box and they try to analyze everything from that perspective. It simplifies things but inevitably there's a lot of missing information. One piece of this information is whether the output of the black box is random. If you can't peer into the black box you can never know if it's random or not random. Thus the social scientists viewpoint doesn't mean any more than someone who isn't a social scientists because most people view the brain as a black box as well.
That being said if there was a field that could closest determine "free will" it would be neuroscience and quantum physics. When I am flicking that switch and attempting to randomize the timing of it all what neurons are firing and what chemical reactions are occurring in my brain? What is the exact mechanism producing this action? Does the randomness of the quantum world leak out of a chemical reaction to influence the macro output of my timing on flipping that switch?
>That is, they would reject the notion that free will is indeed (or at least always) random.
Again they can't reject anything or accept anything related to this. To them, this part of the universe is underneath a black box.
>Sorry again that I have to rely on the statistical view - it is the only one I know a little bit about.
Who says you have to restrict yourself to viewpoints from certain groups? When you restrict yourself to "the statisticians" viewpoint or "the social scientists" viewpoint you are also restricting your intelligence.
Not to talk past each other’s: The definition of causal statistical identification is a formal one, but builds on the axioms of probability theory. The philosophical and even physical underpinnings of these axioms are, as they have always been, part if a heated debate.
However, the system is indeed consistent. And so, in this framework, neither true randomness (which is not a concept that has a definition here) nor independence is mathematically required for causal identification, instead, the switch variable needs to be uncorrelated to unobservables (and so on). That it needs to be correlated to the target variable is obvious, as you say. That is all tautological and simple to show.
I should also note that both independence and correlation are defined in terms of statistical properties that can hold irrespective of whether the world has true randomness. In a way, think if statistics as tying to the world that we see (the data) a consistent way of understanding and inferring. Now, whether the world actually works in terms of measures and moments, well, that is another question. And, I would propose, one that will be difficult to decide, in that it transcends our best ability to work with observational facts.
Note that there are other conceptions of causality and inference, distinct from statistics. I have worked with a group using models without any randomness, relying instead on the modeling of fully deterministic chaotic systems. They can indeed also fit the data, even if they are less practically useful.
As it turns out, a procedure of identifying causality, getting useful results, looks not too different.
So yeah, I think we agree in basic terms. In your view, causality as it is typically applied is fuzzy since its axiomatic framework does not arise from physical, philosophical or perhaps metaphysical facts as you believe or know them to be true. In my view, causality is fuzzy because any pragmatic definition, including one based on experiments, is based on assumptions without which causal identification is impossible. For instance, no matter what experiment you construct, we will need to assume that the operator switching randomly has not been paid money to choose a predetermined sequence by some James Bond villain. Of course, only one of us can be the operator, and so any pragmatic definition of causality (in being communicable) will eventually be fuzzy.
>switch variable needs to be uncorrelated to unobservables
What is an unobservable? Do you have a source for the formal definition of causal identification?
In wikipedia, they immediately skip logics and math and jump to science which indicates to me if a formal definition exists, it's controversial and likely not well known.
Unobservable is any factor that is correlated with outcome and switch, which you do not model or do not have data for.
To identify a causal effect of x on y, you need to exclude the possibility that x is correlated with z and z influences y. If it is so, you need to model z in some way (here, it depends on what we talk about). There are also other considerations, but that is perhaps the most pressing and obvious one. You can quickly convince yourself that the above must hold if a causal effect is to be identified. Under assumptions, for example induced by experimental conditions, the above is also sufficient to identify a causal effect.
The absence of any unmodeled z ( eg the Bond villain in our aforementioned randomness experiment) is essentially always an assumption and not a provable fact, which motivates my earlier discussion. Note also that measuring, modeling or otherwise taking into account any such Z depends on the Z and therefore the data generating process (aka the situation at hand )
Given that causal identification requires assumptions on the DGP, no general or non model based definition can exist.
That is, we CAN say what we mean by causality, however the task of identifying causality has to be stated in terms of a of the data generating process.
Perhaps herein lies the confusion.
Based on the DGP, mathematical definitions of causality do of course exist. For example, in the framework of switch and outcome, as defined, causality is semiparametrically identified if the switching is uncorrelated with other factors that are not modeled. In that case, we identify the qualitative direction of causality, or causal effect. For example, if the switching is true randomness, as per your definition, then the condition is fulfilled. However, randomness is only sufficient not necessary.
What a causal effect is, well, that means depends on the data and situation, for example what values outcome can take and what measures you apply: Expectations, Distributions, Order Statistics and so on.
If we see three people operating switches in terms of dials, we would for example be able to identify the average marginal effect of dialing on outcome. This would be an estimated expectation.
Formal definitions of these concepts can be found in, hopefully, any modern statistics textbook.
Otherwise, there are many formal treatments available. For the concept of statistical identification, Lewbells article „Identification Zoo“ is a good introduction. Formal definitions of identification and causal identification (albeit written for econometricians) are presented.
http://fmwww.bc.edu/EC-P/wp957.pdf
The first sections give a good understanding why it only makes sense to talk about causal effects and causal identification, if we have a model (restrictions on the DGP) in mind.
While the text is generally about statistical identification, you can do the following: Assume that based on your DGP/model, theta_0 (in the text) is the parameter telling you, whether X causes Y. Then, reading section 3 is probably sufficient to get an understanding of what causal identification is.
As it applies to causality and counterfactuals specifically, read Pearl as one framework, and research on Rubin for another. There are now, I think, good books on amazon treating both in more generality, especially outside of econometrics (which is or was for some time leading in non experimental causal analysis). Then, literature on experiments is also concerned with causal analysis, though since the distinction between correlation and causation is less of an issue, the literature has less focus on stating what exactly makes a procedure a causal analysis and what does not.
For that reason I actually recommend looking at the non experimental literature first, since the need for definition arises.
"Why are the lights off?"
"Because I didn't turn them on."
That's not a counterfactual, it's a fact. Just like the admins not configuring file purging is a (presumed) fact in the scenario under discussion in the article. Negative statements are not counterfactuals. Now, they may not be helpful for a 5 Whys analysis, but that's a separate thing.
Counterfactual: If the admins had turned on file purging, then the volume would not have been full.
Why? Because they didn't turn it on so this is a counterfactual in that it is dealing with facts not in existence in the reality under examination and reaching a conclusion of how things would have been different with those different facts. But nearly all the causes in a 5 Whys can be seen as counterfactuals if phrased correctly:
"Builds did not complete." Why? "Kubernetes could not start the pod, and the operation timed out after 1 hour."
As a counterfactual: If Kurbernetes could have started the pod, then the operation would not have timed out after 1 hour and the builds would have completed.