There was this concept of objectives and counter objectives which was meant to combat Goodhart’s law.
Metrics and counter-metrics, IIRC originally coined by Julie Zhuo.
It’s a great framework for making sure that you consider Goodhart’s law, but it’s also only as good as the person thinking about it.
The problem I have seen in practice is that the same person whose job it is to think of the counter-metric (usually the PM) often looks better and gets promoted/paid more when the real costs of chasing the KPI stay hidden. PM leadership doesn’t have the bandwidth to make sure every PM is being rigorous.
I’ve felt this pressure as a PM (I don’t think I gave into the temptation but that’s not for me to judge) and as a developer I’ve seen my PM invest only nominal time on the counter-metric, where arguably you should be spending more time thinking about it than the metric itself. In practice, that resulted in things like security-through-annoyance-and-unreliability.
I’ve been in those shoes, so I know defining metrics to track business goals is really hard. But I do think people on the ground can tell when it’s a matter of difficulty as opposed to a matter of cutting corners or storytelling spin.
Metrics and counter-metrics, IIRC originally coined by Julie Zhuo.
It’s a great framework for making sure that you consider Goodhart’s law, but it’s also only as good as the person thinking about it.
The problem I have seen in practice is that the same person whose job it is to think of the counter-metric (usually the PM) often looks better and gets promoted/paid more when the real costs of chasing the KPI stay hidden. PM leadership doesn’t have the bandwidth to make sure every PM is being rigorous.
I’ve felt this pressure as a PM (I don’t think I gave into the temptation but that’s not for me to judge) and as a developer I’ve seen my PM invest only nominal time on the counter-metric, where arguably you should be spending more time thinking about it than the metric itself. In practice, that resulted in things like security-through-annoyance-and-unreliability.
I’ve been in those shoes, so I know defining metrics to track business goals is really hard. But I do think people on the ground can tell when it’s a matter of difficulty as opposed to a matter of cutting corners or storytelling spin.