Hmm, do they? The logical reasoning test in that page is a question about lab rat studies on coffee+birth defects, and a hypothetical spokesperson's response that they wouldn't apply a warning label because the government would lose credibility if the study were to be refuted in future. You're then asked a multiple choice question:
1. Which of the following is most strongly suggested by the government’s statement above?
(A) A warning that applies to a small population is inappropriate.
(B) Very few people drink as many as six cups of coffee a day.
(C) There are doubts about the conclusive nature of studies on animals.
(D) Studies on rats provide little data about human birth defects.
(E) The seriousness of birth defects involving caffeine is not clear.
Given the structure of this question I assumed there'd be more than one right answer but apparently, the only "logical" answer is C.
Maybe the word logic is used differently in the legal profession, but this doesn't resemble the kind of logic test I'm used to. It's about unstated/assumed implications of natural language statements i.e. what a 'reasonable' person might read into something, rather than some sort of tight reasoning on which logical laws could be applied. I can see why that's relevant for lawyers but it's not really about logic.
Still, let's roll with it. (A) and (B) are clearly irrelevant given the stated justification, strike those. But (C) and (D) appear to just be minor re-phrasings of each other. Why is C correct but D not? An implied assumption of the study is that rat studies provide a lot of data about human birth defects, and the government's position implies that they don't agree with that. D could easily be a reasonable subtext for that position. E could also be taken as a reasonable inference, that is, the government believes there's a risk the study authors are using an exaggerated definition of birth defect that voters wouldn't agree with, and that 'refutation' of the study would take the form of pointing out the definitional mismatch.
So if I was asked to score this question I'd accept C, D or E. The LSAT authors apparently wouldn't.
That said, the "analytical reasoning" sample question looks more like a logic test, and the logic test looks more like a test of analytical reasoning. But even their bus question is kind of bizarre. It's not really a logical reasoning test. It's more like a test to see if you can ignore irrelevant information. The moment they say rider C always takes bus 3, and then ask which bus {any combination + C} can take, the answer must be (C) 3 only. Which is the correct answer.
> I do not think that it really matters which questions you choose as long as they span a wide enough difficulty range so that you are able to separate participants.
The problems here are pointing at a fundamental difficulty: all claims about competence/expertise are relative to the person picking the definition of competent. In this case the tasks are all variants on "guess what the prof thinks the right answer is", which is certainly the definition of competence used in universities, but people outside academia often have rather different definitions.
So the questions really do matter. If the DK claim was more tightly scoped to their evidence - "people who think they're really good at guessing what DK believe actually aren't" - then nobody would care about their results at all. Because they generalized undergrads guessing what jokes Dunning & Kruger think are funny to every possible field of competence across the entire human race, they became famous.
> Given the structure of this question I assumed there'd be more than one right answer
I did not, since the question is explicit about there being one correct answer only:
“Which of the following is most strongly suggested by the government’s statement above?”
> but this doesn't resemble the kind of logic test I'm used to. It's about unstated/assumed implications of natural language statements
Agreed.
> But (C) and (D) appear to just be minor re-phrasings of each other.
I think the key here is “there are doubts”. The government’s position stems from doubts on the conclusive nature of the study, that’s it. The statement doesn’t say anything about how much data studies on rats provide about human birth defects. If we’re being logical, studies on rats provide “no data” on human birth defects. Across many studies with different substances there may be a correlation (p(human birth defect | rat birth defect) = x), but an observation of birth defects on rats for a particular substance gives us data about rat birth defects, not human ones.
Ah yes - is vs are. You're right. I think I assumed there'd have to be >1 right answer after reading the options.
It's a remarkably poor question, but option (C) isn't about doubts on the conclusive nature of this specific study, but rather the nature of all studies on all animals. You could credibly argue (and I'd hope a lawyer would!) that no government would base policy on doubting all animal studies and that their position in this case must therefore be due to something about this specific study, e.g. the usage of rats, or the topic of birth defects, or both. So they could argue that (D) is the most logical answer.
Not that it really matters. Pretty clearly the LSAT authors are using the word logical in the street sense of "makes sense" or "sounds plausible" rather than meaning "based on an inference process that's free of fallacies". If DK based their test of competence on questions like this then it doesn't mean much, in my view.
If the validity or significance of the paper depends on whether LSAT questions are fit for DK's purpose, we have entered a much more subjective realm than whether they mishandled the statistical analysis - but as we are there, now, I feel that this particular question is not as bad as it is being portrayed.
Firstly, I think we should put aside the fact that it is labeled as a test of "logical reasoning": it is certainly not a test of formal logical reasoning, and an ambiguous or erroneous label does not necessarily make it a bad question (it is not necessary that it be characterized at all.)
Secondly, we are not logically obliged to accept that it has only one answer among the options presented, though if it has more or less than one while the people who posed it thought it had exactly one, that is a problem (I once was nearly expelled from a class for making this point at greater length than the instructor liked!) On the other hand, the question asks which of the candidate answers is most strongly suggested by the passage, which is not a statement that the others are false.
Here, however, it certainly has no more than one answer among the candidates: there is nothing in the passage that has any bearing on options A, B, D or E
- this is perhaps most obvious in the case of B, but the others are like it. In particular, with respect to D, that specific issue is not raised, and furthermore, if it was the government's opinion now that D was the case to the extent of having a bearing on the decision, there would be no need to explain its position in terms of a potential future determination that the tests are inconclusive.
C, on the other hand, is suggested by the government's explanation: if the tests were conclusive, their future refutation would not be a worry.
As I said, this is not a test of formal logic, where the government's response would not imply the possibility of future refutation. Nevertheless, to explain something on the basis of a premise that is only formally possible would be almost as much an informal fallacy as begging the question, IMHO, and one might suspect it is being offered deceitfully (a concept that has no place at all in logic.)
The sort of analysis of natural language called for here (to see what are and are not the issues being considered) is useful and important, for lawyers and the rest of us, and it is, as I have set out above, more objective than "makes sense" or "sounds plausible." If people were more practiced in analytical reading, then corporations, governments and other organizations would less easily get away with blatant non-sequiturs in their explanations of their positions and actions ("there is no evidence the attackers took any personal or confidential information"...)
The second question labeled analytical reasoning is probably closer to what you consider a logical reasoning question, maybe they picked questions more like those?
That's the issue - we don't actually know what they did. Which means their claims would have to be taken on faith.
Now, maybe other researchers designed different more rigorous studies that are replicable and which show the same effect. That could be the case. The point I'm making here is that the DK paper isn't by itself capable of proving the effect it claims, and that you don't need a statistical argument to show that. Sanity checking the study design is a good enough basis on which to criticize it.
1. Which of the following is most strongly suggested by the government’s statement above?
(A) A warning that applies to a small population is inappropriate.
(B) Very few people drink as many as six cups of coffee a day.
(C) There are doubts about the conclusive nature of studies on animals.
(D) Studies on rats provide little data about human birth defects.
(E) The seriousness of birth defects involving caffeine is not clear.
Given the structure of this question I assumed there'd be more than one right answer but apparently, the only "logical" answer is C.
Maybe the word logic is used differently in the legal profession, but this doesn't resemble the kind of logic test I'm used to. It's about unstated/assumed implications of natural language statements i.e. what a 'reasonable' person might read into something, rather than some sort of tight reasoning on which logical laws could be applied. I can see why that's relevant for lawyers but it's not really about logic.
Still, let's roll with it. (A) and (B) are clearly irrelevant given the stated justification, strike those. But (C) and (D) appear to just be minor re-phrasings of each other. Why is C correct but D not? An implied assumption of the study is that rat studies provide a lot of data about human birth defects, and the government's position implies that they don't agree with that. D could easily be a reasonable subtext for that position. E could also be taken as a reasonable inference, that is, the government believes there's a risk the study authors are using an exaggerated definition of birth defect that voters wouldn't agree with, and that 'refutation' of the study would take the form of pointing out the definitional mismatch.
So if I was asked to score this question I'd accept C, D or E. The LSAT authors apparently wouldn't.
That said, the "analytical reasoning" sample question looks more like a logic test, and the logic test looks more like a test of analytical reasoning. But even their bus question is kind of bizarre. It's not really a logical reasoning test. It's more like a test to see if you can ignore irrelevant information. The moment they say rider C always takes bus 3, and then ask which bus {any combination + C} can take, the answer must be (C) 3 only. Which is the correct answer.
> I do not think that it really matters which questions you choose as long as they span a wide enough difficulty range so that you are able to separate participants.
The problems here are pointing at a fundamental difficulty: all claims about competence/expertise are relative to the person picking the definition of competent. In this case the tasks are all variants on "guess what the prof thinks the right answer is", which is certainly the definition of competence used in universities, but people outside academia often have rather different definitions.
So the questions really do matter. If the DK claim was more tightly scoped to their evidence - "people who think they're really good at guessing what DK believe actually aren't" - then nobody would care about their results at all. Because they generalized undergrads guessing what jokes Dunning & Kruger think are funny to every possible field of competence across the entire human race, they became famous.