> I've always wondered, how do you test your own intelligence?
He is not measuring intelligence; He's measuring reaction times (how long he takes to solve a very simple arithmetic problem; or how long he takes to see which letter in a 4 letter string appears twice) and fine balance/motor function (how long he can keep his balance on an unstable platform he has built)
He went over these things in his blog in great detail over the years. He keeps trying new tests, keeps using those that are statistically stable in a "stable" environment (without changing nutrition / sleep / etc, and after a learning period) - and after he has a statistically significant baseline, he measures how e.g. eating more butter or flax oil changes the results of these tests.
Yes, this is drawing conclusions for n=1, but for himself, these conclusions are very scientific - and us humans are not unique enough for them to have wider applicability.
No, you cannot deduce from n=1 to the entire human race. Yes, his conclusions are plausible and most of them are verified by others. Read e.g. gwern.net on vitamin D.
I'm sorry, but science is neither "we measure the things" nor is it "we look for the cause." Yes, those are things which scientists do, but no, that is not the essence of science.
There is no attempt being made to come to any wholesome understanding. That is, a scientist is trying to model nature, and this consists in two parts: (a) develop a model, (b) test it against nature. Seth Roberts has perhaps hit upon a way to develop models, but he doesn't seem to test them. That is, cum hoc, ergo propter hoc -- "with this, therefore because of this" -- is not actually a test for causation, but merely a guess for causation.
To do a proper test, you need variable isolation, and the stability of test results is no guarantee that it is an isolated variable. A good example of an isolated variable is a simple light switch. A terrible example of an isolated variable would be your time of waking up, because that time cannot be isolated from your own thoughts and beliefs -- that is, many people, especially if they're not sleep-deprived, can wake up earlier simply by telling themselves "I'm going to wake up early tomorrow." (I myself usually wake up before my alarm clock goes off.)
Humans in general are not light switches. As has been documented repeatedly in medicine, dual effects of placebo and hypochondria plague us; things we expect to be medicinal often placate us even when they have no medicinal content and you can suddenly feel the symptoms of a malady soon after reading about it on Wikipedia.
So even if you want to say "for himself, these conclusions are very scientific," you're going to have to account for the problem that he is probably going to confirm whatever he already expects. That is, any "follow up" tests after the first guess are already tainted by the fact that Seth knows what's being tested.
> There is no attempt being made to come to any wholesome understanding
Do you actually follow Roberts, or is your understand based on this one short summary? Because, e.g. "What Makes Food Fattening" (77-page pdf here http://media.sethroberts.net/about/whatmakesfoodfattening.pd... ) is very much an attempt to come to wholesome understanding, develop a model, test it against nature. (The test is "putting it out in the wild", and the result is "works beautifully for 80% of people who try, not at all for the remaining 20%". He's not funded to test this, nor is he making any money of it - I don't think there's a better route for him to take).
> To do a proper test, you need variable isolation, and the stability of test results is no guarantee that it is an isolated variable
Yep. Except, in the real world, NO published result related to nutrition, and almost no published result regarding medicine, is a proper test with isolated variables. Including almost every guideline your doctor works by.
> that is, many people, especially if they're not sleep-deprived, can wake up earlier simply by telling themselves "I'm going to wake up early tomorrow." (I myself usually wake up before my alarm clock goes off.)
That is true. And yet, a lot of people want diets to work, and they don't. A lot of insomniacs want a placebo to work, and it doesn't. Placebo is powerful, but is not all-powerful. It is stronger with some people, weaker with others.
Discarding results just because they were not the result of a double-blind placebo-controlled test is not rational. Neither is accepting results just because the author thinks that they are double-blind placebo-controlled:
e.g., almost all placebos today are sugar pills; If the tested-against-material has a side effect, such as causing flushing or a dry tongue (though does not produce the wanted outcome -- which is very often the case), this is not in face a double-blind; the patient knows that they did not get a placebo, and the whole test is useless. Yet, this is the gold standard.
Furthermore, if you read the NEJM / BMJ / Nature / Science medical papers, you'd notice that their result are tested on (and are therefore only valid for) a small ethnic group, a small age group, etc. But then, very unscientifically, it is assumed (by others -- usually not by the authors of said paper) to apply much more widely.
> That is, any "follow up" tests after the first guess are already tainted by the fact that Seth knows what's being tested.
Yes, that does not make the results any less true or useful: Butter makes seth faster at arithmetics, in a consistent and statistically significant way. That may not be true for others. And may be entirely a placebo. Nevertheless, if Seth wants to be faster at arithmetics, he can do that by eating butter, regardless of the cause. Is that not science?
(1) Indeed, I am commenting on this particular mode of understanding, where one keeps a journal, waits for a deviation of a test from normal, and then tries to correlate the journal with the deviations after the fact.
(2) "NO published result related to nutrition" is overbroad. Obesity for example is clearly related to nutrition and there are many published results where many variables which could affect obesity are quite well-isolated -- twin studies to isolate genetics, studies of how obesity rates vary based on physical location in various cities, and so forth.
(3) I aim to "discard the results" only insofar as they claim that they have measured an agentic relationship, which these results have indeed claimed to measure. Seth's official conclusion is that this was "data that suggested butter improved my mental function." If the variable is not isolated, then attributing the agency to butter is worthless. It might have been an otherwise random variation at around the same time as he switched to butter -- the data he's describing has a 40ms variance as well as certain long-term patterns and the effect that he's describing is a 30ms improvement, so it's quite possible that in an autoregressive model you would just see one substantial "step down" of 80ms which doesn't get compensated due to hysteresis. Or perhaps the correlation is indeed correct but wrongly attributed -- perhaps Seth has not noticed but he tends to use butter to fry peppers and pork fat to fry meats, and now that he is using butter his diet contains 2% more vegetables. Perhaps now that he uses more butter, he eats more toast and gets 2% more fiber in his diet, without having written that down in his journal.
The problem is precisely the word "makes" in your sentence "butter makes Seth faster at arithmetic." You have absolutely no evidence that it's the butter which is making Seth faster at it. And that is why the conclusion "butter makes Seth faster at arithmetic" is not a scientific conclusion.
> Obesity for example is clearly related to nutrition and there are many published results where many variables which could affect obesity are quite well-isolated -- twin studies to isolate genetics, studies of how obesity rates vary based on physical location in various cities, and so forth.
Care to show me, for example, a single nutrition study where "skin color of participant" was a variable controlled for? It is known that cholesterol metabolism (eventually into) Vitamin D is greatly dependent on skin color -- and yet, it does not appear in a single nutrition study, despite the greatly known significance of both serum cholesterol and serum vitamin D.
I'm taking this ad absurdum in an attempt to show you that if you apply this standard rigorously (as one should), then no result really holds up.
And by the way, "obesity is clearly related to nutrition" is only true in the sense that "everything is related to nutrition". There are no results of things that affect obesity that are "quite well isolated" in humans that I'm aware of.
> Or perhaps the correlation is indeed correct but wrongly attributed -- perhaps Seth has not noticed but he tends to use butter to fry peppers and pork fat to fry meats, and now that he is using butter his diet contains 2% more vegetables ...
If you actually followed him, you'd know that's not the case. He is very diligent about isolating variables as much as possible, and thoroughly documenting his day, including food intake, how much TV he watched, etc -- and the butter hypothesis was actually an attempt to strengthen a hypothesis having to deal with animal derived fats (which among other things, included bacon). It wasn't just observational - it was part of a variable isolation experiment, that (within limits) was comparably rigorous to any other non-blinded test (which are far more common than you'd think; in fact, many if not most supposedly double blinded tests aren't).
> You have absolutely no evidence that it's the butter which is making Seth faster at it.
He has much more evidence for than you'd expect from this short posting, if you care to look at it. This conclusion is indeed supported by data, and is scientifically valid.
> I'm taking this ad absurdum in an attempt to show you that if you apply this standard rigorously (as one should), then no result really holds up.
Then you should be willing to do what happens when a reductio ad absurdum fails: give up and admit you are wrong.
> And by the way, "obesity is clearly related to nutrition" is only true in the sense that "everything is related to nutrition".
Uh, no. Obesity is a nutritional disorder, as distinct from other things like having cats, which are neither caused nor hampered by good nutrition. What the hell are you smoking?
> He has much more evidence for than you'd expect from this short posting, if you care to look at it. This conclusion is indeed supported by data, and is scientifically valid.
I searched his website and all I could find was one particular crappy-looking graph with a bunch of discussion about his responses to vague questions, and an ad-hoc explanation (a competing omega-3 deficiency) when the data did not fit his expected pattern.
Thanks. For some reason, I was unable to find this when I last looked (might have been before this came out).
> Uh, no. Obesity is a nutritional disorder, as distinct from other things like having cats, which are neither caused nor hampered by good nutrition. What the hell are you smoking?
Are antibiotics with no nutritional value considered nutrition these days? There are hundreds of substances that seem to cause obesity independent of nutrition (that is, when isolated as a variable compared to nutrition), most prominently psychiatric drugs, but -- as is shown by this study -- also antibiotics, which are supposed to be "nutritionally inert".
> I searched his website and all I could find was one particular crappy-looking graph with a bunch of discussion about his responses to vague questions, and an ad-hoc explanation (a competing omega-3 deficiency) when the data did not fit his expected pattern.
He did not yet bother summarizing these results (the way he did for his weight loss theories, in http://media.sethroberts.net/about/whatmakesfoodfattening.pd... - but there's a lot more discussion of it through the years) , I suspect that it is because he does not believe he nailed anything yet. However, he asks people to "try at home", and several of his results (mostly regarding dietary fat, but also about mood and sleep) have been confirmed by other people as well.
He's under no "publish or perish" stress like most papers you see, so he isn't trying to get anything production quality. I've been following him for 7 years now, and while it isn't blinded or double blinded, it's otherwise as good as (and usually better) than observational studies I find in top rated journals.
He is not measuring intelligence; He's measuring reaction times (how long he takes to solve a very simple arithmetic problem; or how long he takes to see which letter in a 4 letter string appears twice) and fine balance/motor function (how long he can keep his balance on an unstable platform he has built)
He went over these things in his blog in great detail over the years. He keeps trying new tests, keeps using those that are statistically stable in a "stable" environment (without changing nutrition / sleep / etc, and after a learning period) - and after he has a statistically significant baseline, he measures how e.g. eating more butter or flax oil changes the results of these tests.
Yes, this is drawing conclusions for n=1, but for himself, these conclusions are very scientific - and us humans are not unique enough for them to have wider applicability.
No, you cannot deduce from n=1 to the entire human race. Yes, his conclusions are plausible and most of them are verified by others. Read e.g. gwern.net on vitamin D.