I enjoyed the quiz, but I am not a big fan of the positive_effect/no_effect/negative_effect classification, because it hides the error of the measurement.
Instead, I propose to use a slider, like:
> "In your estimate, how does the second response program influence the rate of repeat offenses?"
> Underneath, a slider on a one-dimensional axis, with appropriate labels. For example, on the leftmost side: "50% increase in repeat offenses", and on the rightmost side "50% decrease in repeat offenses".
> Clicking the "reveal" button superimposes the measured effect and its errorbar (this is important!) onto the slider.
That way everyone can see if the measured effect is relevant, or if the measurement error is so large that the study might have missed a relevant effect.
The downside of a slider is that the slider range will influence ("prime") the responses. But I doubt that it would influence the direction of the guess, so you could still extract useful information out of the reponses.
That would be better, though quite a bit more effort!
Usually these studies only have enough resolution to say "probably worked", "no measurable effect", "probably negative", and even then are often wrong.
Instead, I propose to use a slider, like:
> "In your estimate, how does the second response program influence the rate of repeat offenses?"
> Underneath, a slider on a one-dimensional axis, with appropriate labels. For example, on the leftmost side: "50% increase in repeat offenses", and on the rightmost side "50% decrease in repeat offenses".
> Clicking the "reveal" button superimposes the measured effect and its errorbar (this is important!) onto the slider.
That way everyone can see if the measured effect is relevant, or if the measurement error is so large that the study might have missed a relevant effect.
The downside of a slider is that the slider range will influence ("prime") the responses. But I doubt that it would influence the direction of the guess, so you could still extract useful information out of the reponses.