I've heard similar statements in the past, but I think I lack the statistics knowledge to make sense of it. Would you mind expanding on that a little?
I.e., suppose that 3 of the questions were meant to test some theory that the researchers have, and the 21 other questions were thrown in for the reasons stated above.
Now suppose that the answers to the 3 seriously-chosen questions weren't predictive of long life, but 5 of the other 21 questions were predictive of long life.
It sounds like you're saying that somehow that 5-question correlation is bogus, but the 3-intentional-question correlation would be legitimate. But the only difference I see between the two groups is subjective: whether or not the researchers anticipated the questions being meaningful. I don't see how/why that would be relevant to making inferences about the wider population.
The danger is similar to any backtested algorithm. You can always go back and find theories to fit your data. That doesn't mean they will be predictive going forward. It also doesn't mean they _won't_ be predictive, and I would say they can form a good hypothesis for a new experiment. It's not that it's meaningless, it just lacks the power of scientific evidence. It's comparable to a non repeatable experiment, where the hypothesis counts as a first "repetition".
At least that's my interpretation as a non statistician, but I think there's a difference between statistical significance and scientific proof. The findings may be statistically significant without actually proving anything.
The problem is that the entire premise of the study is flawed, whether three questions were asked, or a hundred. Asking a hundred questions simply means it’s more likely your study will “succeed” in the sense of finding something to report that aligns with your agenda. Statistically, take any sample size. If you measure enough factors, you are bound to find some bias in your sample size.
Instead, a better study would consider actual science. Microbiology, biochemistry, etc and form measurable hypotheses instead of this spray-and-pray pseudoscience
I.e., suppose that 3 of the questions were meant to test some theory that the researchers have, and the 21 other questions were thrown in for the reasons stated above.
Now suppose that the answers to the 3 seriously-chosen questions weren't predictive of long life, but 5 of the other 21 questions were predictive of long life.
It sounds like you're saying that somehow that 5-question correlation is bogus, but the 3-intentional-question correlation would be legitimate. But the only difference I see between the two groups is subjective: whether or not the researchers anticipated the questions being meaningful. I don't see how/why that would be relevant to making inferences about the wider population.