I believe this recent paper demonstrates a method for allowing these large language models to perform this "sanity check" automatically[0].

[0]: Self-Consistency Improves Chain of Thought Reasoning in Language Models https://arxiv.org/abs/2203.11171