These results from observational studies almost never replicate in randomized studies. It's because people who decide to take glucosamine (or not) also do lots of other confounding things, like visit the doctor more, or have higher salaries, or don't live with people who smoke etc. It's been shown many times you can't really adjust for these things. Epidemiologists love doing these studies, because they are comparatively easy. The hardest part is the data sharing agreement with whatever registry you are using. You don't even need to leave your desk to collect any data.
Because academics and primary consumers of these studies understand their uses and limitations. Not all science has to be groundbreaking new discoveries, simply trying to replicate an existing study or exact same methodology on different cohort is perfectly fine.
The problem is when it gets printed in mainstream press who sensationalize and then posted to reddit/hn where people go 'duh this is obvious, this is a dumb thing to study', 'sample too small', 'not controlling for income/confounding'.
One answer: anyone can start a journal, there are no laws or standards. Peer reviewers are busy and don't always have the background to criticize every aspect of a paper.
Another: not all research can or should have randomized studies (like it just wouldn't make sense if you were, say, determining the structure of a protein). So drawing the line as to what does and doesn't need a specific approach is blurry, and really, can only be enforced during the peer review process, or by the editor.
On the other hand, this kind of work can still serve as a hypothesis generator. Someone else will read this, think, "well they did this all wrong!" and perform a better study that clearly demonstrates whether it has an effect or not.
I guess it's ok to publish uninteresting and probably false things. This could be a fairly accurate description of biomedical science to be honest. It's annoying when it ends up on HN for example, or in the health section of a tabloid newspaper.
In theory, these observational findings could suggest treatments that should be evaluated in a randomised trial. I am not actually sure this is true however, it is possible they are less useful than selecting interventions for further testing in some other way. Anecdotally, folk remedies may have a higher hit rate for selecting treatments for proper evaluation. This is often surprising - eg a precursor to aspirin is found in the Willow tree, which has been used as a medicine for thousands of years. Another example is the foxglove plant from which the heart drug digoxin is derived, which was used as a herbal medicine predating modern medicine.
Not questioning the need for RCTs, but they have their own issues. Like, let's say you do a RCT and it shows an effect, but then in a large epi observational study there isn't any? There would be some problems to work out.
RCTs are great but can have problems with generalizability to real-world conditions. Ideally you'd study both.
With a sufficiently robust RCT, I'd think that indicates an issue with the observation in the observational study -- i.e., there are some confounding factors in the real world -- rather than any sort of issue with whatever effect the RCT found.
Conversely, if we had a large observational study that was then contradicted by a robust RCT... well, I'd still be inclined to trust the RCT.
(At the same time, I think a good observational study is, by its nature, more applicable to one's own life. If an observational study suggests that $FOOBAR is good, then -- as long as there's also a solid RCT confirming that $FOOBAR is safe -- why not try out some $FOOBAR for yourself?)
This is a good point. I was recently having a conversation with a non-scientist (I am not one either) about two different studies. One was an RCT that found no effect for a treatment, and the other was a "case control, test-negative" design study that found a beneficial effect. My friend said the latter was as good as an RCT because it also has a control arm.
But when I looked up what "case control, test-negative" means, I found that it is observational, and there is no intervention provided to any of the individuals. Some of the recent discussions of the design indicate that it is understood by scientists, but can mislead laypeople, who assume that its conclusions are much more robust than they actually are.
We should definitely continue to do research of many types, partly as a way to figure out where to spend the time/money to do RCTs (which are more expensive than other types of analysis). But we need people reporting on studies to be very clear up-front when they are not describing an RCT. They should say what the study's conclusion means, versus what it would mean if there were a similar 'finding' in an RCT. Otherwise people will not understand that they are being told a weak conclusion, not realizing there is a strong conclusion that has gone unmentioned.
All science needs to be published. Pilot studies, observational studies, quasi-experimental studies. Otherwise, we don't have the information we need to create randomized control trials.
However, I'd strongly support keeping all that science from being publicized.
You can't do randomized studies for things like nutrition if the question is "does eating red meat for a decade increase the risk of cancer". Nobody's going to comply with it.
Not everything can be done with a randomized controlled study. For some fields (quantum mechanics, volcanology, climate change, paleontology...) most studies can't be randomized, much less blind. This is also true of longitudinal studies: you can't fully control anyone's exercise and diet over months, much less decades.
Yet we want to do research in these areas. So we have to make do with what's possible.
True! But medicine just so happens to be one of those fields that lends itself well to RCT’s. Perhaps if the OP had reworded their statement to everything in medicine it might make more sense.
Well no, I’m not wrong. That article you linked is a review of reviews comparing RCTs and observational studies for medical interventions, primarily surgical studies or pharmacological treatments with well characterised effects. Observational studies of supplements and nutritional interventions on the other hand have a very poor track record of RCT validation.
Such a review is necessarily limited to topics which have both an RCT and an observational study.
As such, the general bias against publishing non-significant results could mean the conditions for being able to even try such a review risk biasing the review towards exaggerating agreement.
(It also seems the Cochrane review here may itself only be considering other reviews - a metameta analysis of sorts – which might also be influenced by a selection bias, if there’s any chance at all that researchers choosing work, & then publication decisions, prefer to address questions where there’s more agreement, rather than less-attention-catching mixed results.)
The most convincing evidence here would be how strongly preregistered RCT results, whether ever published or not, tend to match earlier observational studies. It doesn’t appear to me that this Cochrane review is focused on that.
@gojomo: Isn't it normally the case that an RCT follows an observational study? There's not much point in doing an observational study if someone's already done an RCT, but there is the other way round, since RCT's are more trusted.
Wrt publication bias; an underestimation of the false positive rate of observational studies wrt RCTs could occur if non-significant RCTs are being held back at a higher rate than observational ones, but it seems more likely to me that it's the other way around since RCTs are generally more expensive and time consuming than observational studies, and more trusted, so it's a bigger loss if the result isn't published.
I don’t have a sense of the ‘normal’; it might vary by field.
For example, for COVID vaccines, we’re being inundated with observational data - often informal, sometimes in reviewed studies - long after initial approval RCTs.
Hmm, that would be interesting, except it seems they only considered 14 studies. They find in 11 of these observational studies match the results of RCTs.
Im sure this will be different in areas where there is no public awareness of a risk factor vs ones where there are. Eg healthy diets select for people who care about their health, so it's not so infomative, vs something where the link is not publicly known (so that the observational study is already somewhat blind/randomized).
Good advice in general but it’s worth reading to see what confounders they tried to correct for. There’s still information to be gleaned.
This part caught my eye:
> A stronger association between glucosamine use and decreased lung cancer risk was observed in participants with a family history of lung cancer when compared with those without a family history.
This was done in Guangzhou where I lived for years until 2018. The incidence of smoking amongst males still higher - gut guess of 30-40% adult males are smokers in the city and province. So in their cohort one would expect to be seeing still living relatives of young adult smokers who are also smokers.
Interesting that it’s sold as supplement for joint and related tissue repair and yet first big science study with observable results is about cancer.
Other interesting point is that it’s study of cancer. And that is simply because China still has huge numbers of smokers and the health cost is stratospheric.
Is it actually stratospheric though, when you take out QOL year adjustments? Other studies, such as this one in Finland[1], found that when you don’t take into account QOL adjustments for the smoker that smoking is actually a net benefit for society because the smokers die earlier and don’t draw on the very expensive late life healthcare.
To me it is plausible that in a time of great social and economic change, the odds of behavioral confounds in observational studies might increase. Non-stationarity everywhere.
That said, a meaningful result here would be welcome.
You’re right that there is non-random allocation to the exposure (taking glucosamine), but this is something that can be addressed to an extent with a bit of statistics.
I was disappointed to see the authors didn’t take the additional step of doing propensity score matching or weighting to account for propensity to take glucosamine.
Ben Goldacre's book 'Bad Pharma' has a good discussion around this, especially in terms of caveats and gotchas to watch out for with such studies.
It might be more 'lay person' than you are looking for, but it's an excellent read for myriad other reasons, so you wouldn't be wasting your time (IMO).
Thanks. I was hoping for something online, like a blog post/website, which is an easy way to spread the word to other people so they know how to contextualize these things.
I agree with that. You can see this in Ontario COVID statistics.
People that did not take the vaccine, despite all the pressure, tend to be healthier as population. They kind of self selected by being self confident in their health and not worrying about dying from COVID. The triple vaccinated in Ontario are now the highest risk of getting COVID. They too self selected, by tending to be older with pre existing health problems, and worried enough to get boosters.
> The triple vaccinated in Ontario are now the highest risk of getting COVID
I don't understand how the source you shared supports this claim*. If anything, it shows the opposite - you're at far higher risk of death if you're unvaccinated. And that's after considering the fact that unvaccinated people are likely to be younger and fitter.
The unvaccinated have, at best, half as many deaths per 100k as those who are fully vaccinated + booster (in reality, the two lines mostly track each other). Which sounds good, until you realize that the unvaccinated make up only 9% of the population. 33% of the deaths coming from 9% of the population - healthier, they are not. Delusional, would be more correct.
* I'm assuming "getting COVID" here means "having serious consequences from an infection" as opposed to merely being infected. The vaccinated make up the vast majority of the population, so it's obvious that they are also at the highest risk of being infected by COVID. There are very few unvaccinated people left for the virus to infect.
The numbers have already been adjusted. So what you're seeing is the rate of covid cases per 100k people in each group. 3rd graph from the top. You're trying to adjust the numbers twice for the unvaccinated.
If you don't believe me, here is a second set of data from Walgreens. Look at the third page.
Doesn't it? I see a graph titled "Deaths involving COVID-19 by vaccination status". And that still shows higher deaths/100k in the "Not fully vaccinated" group for every given adult age range: 18-39, 40-59, and 60+ (look at the All time data).
> here is a second set of data from Walgreens. Look at the third page.
That shows the positivity rate. I'm not talking about the positivity rate. We're beyond that now. Everyone will get Covid someday. The vaccine protects you from dying or having serious health issues. There's no vaccine, for any disease, that can prevent you from contracting the disease. That would require vaccines to create force fields around your body, which are science fiction. In the case of the most efficacious vaccines, your immune system will be so well-prepared that your body will fight off the infection without you ever getting any symptoms.
I missed the "Deaths stats" as I did not see those before, the last time I looked.
So the last cited death rate is 0.01 vs 0.02 / per 100,000 people. So the vaccinated have 1 death per (100 * 100,000 = 10 million). Or 1 death per 10 million and the "not fully vaccinated" have 2 deaths per 10 million. To give context Ontario has a population of 14 million people. This seems like an easy relative win for the vaccine.
Except:
1) The numbers of deaths are so low, its kind of meaningless to extrapolate to the general population from them, because of sample bias effects.
One reason being that deaths could be coming from a very likely specific sub population. For example very old sick people already in hospital or nursing homes near death that contract the disease. Pretty much anything could kill them. It has no bearing on how a random person from the general population would react. You might have situation where for example there are people that chemo therapy failed, and they either refuse the vaccine (since they will die within weeks anyway) or might be so sick and too weak to take the vaccine and then contract covid as the last straw that breaks them.
So it would be very disingenuous to claim based on such small number that the vaccine lowered deaths in the general population. It would be like telling people in Hawaii to wear gloves to prevent frostbite, based on data collected from Canada showing that people that did not wear gloves had twice the rate of frostbite.
2) "Unvaccinated" and "Not Fully Vaccinated" are two different things according to their definition. They're clearly counting anyone that died within two weeks of getting a shot as being unvaccinated, even if it was their second shot. This is not a fair comparison.
3) Our prime minister caught covid twice in the last 4 months despite being vaccinated and boosted.
There have been studies that show that natural acquired immunity is longer lasting. Since the odds of dying are so low at this point, would you rather get covid once and feel a little bit sicker, unvaccinated, for longer lasting immunity. Or would you rather get vaccinated and boosted and catch it twice, feeling a little less sick each time.
Which is what the data seems to be showing is happening.
This seems like it should be personal preference decision.
Last update was June 10. So it's still pretty recent.
Ontario did not release data on death rates to the public. I don't know why.
I briefly read on twitter that UK is showing startling differences now, with the vaccinated having a much higher all cause mortality rate. If true, I don't know if its this effect at play or if some of the conspiracy theories were right.
You are right that people who take vaccines and who do not probably bifurcate along other lines as well. But you’re doing everything even more of a disservice by speculating the dimensions as if it’s established fact. Maybe the unvaccinated aren’t healthier they’re just liars? Also any self respecting study would control for the simple demographics like age so your hypothesis would be quite moot if the study was done with any rigor.
It was literally never claimed that it was 97% effective at stopping COVID. The Pfizer vaccine was 97% effective at preventing severe disease after two doses in early 2021. Don't bring conspiracy nonsense into every discussion about healthcare.
Here is the claim from pfizer's own press release.
"Vaccine effectiveness was at least 97% against symptomatic COVID-19 cases, hospitalizations, severe and critical hospitalizations, and deaths. Furthermore, the analysis found a vaccine effectiveness of 94% against asymptomatic SARS-CoV-2 infections. "
You just proved yourself wrong? What do you think this says? It literally never claims to be 97% effective at (your words) stopping COVID. That implies that it prevents the infection from spreading and/or causing symptoms, neither of which is part of that statement in the press release.
To be very specific they took two samples of about 22,000 people each. One vaccinated and one given placebo. Then they counted how many people they caught with covid in each sample. 8 vs 162. Then based on those numbers claimed the 97% effectiveness as stopping infection (people getting sick).
Here's a video confirming that this is how they measured efficiency.
8 vs 162 is the people that they confirmed with PCR test to be positive.
They did not actually test everyone for covid. They admit that there were other people they suspected of having covid.
"Among 3210 total cases of suspected but unconfirmed COVID-19 in the overall study population, 1594 occurred in the vaccine group vs 1816 in the placebo group." Page 42 of the pfizer data leaks.
Hope you see why claiming 97% efficiency to the public was misleading.
After a year, and millions of vaccines later, we can clearly say that the study was bullshit. The vaccine did not offer anywhere near the sort of long lasting immunity that approaches 97%.
Our prime minster despite having two shots and boosted was sick with covid twice in 4 months.
So to cite this study as being accurate is kind of like claiming pigs can fly. Yes I suppose they could. If they jump off a cliff, for a brief second they could.
Again, you are completely wrong and simply spouting vaccine conspiracy nonsense. It is very clear from the study that those "suspected but unconfirmed COVID-19 cases" are those who reported one flu-like symptom but had a NEGATIVE PCR test. A very detailed explanation of why you are wrong to want to include those in the efficacy percentage number can be found here: https://www.covid-datascience.com/post/refuting-peter-doshi-.... It is not clear whatsoever that the study was bullshit, and the results speak for themselves. An anecdote about your Prime Minister is meaningless and occurred AFTER this initial study period when the variants had changed the landscape and these numbers were no longer applicable.
Since you are still claiming that it was 95% effective. Can you give a time estimate for how long that efficiency lasted?
This study concludes " Primary immunization with two doses of ChAdOx1 nCoV-19 or BNT162b2 vaccine provided limited protection against symptomatic disease caused by the omicron variant. "
I read the article you cited. Here's the problem as I see with it.
"Among 3210 total cases of suspected but unconfirmed COVID-19 in the overall study population, 1594 occurred in the vaccine group vs 1816 in the placebo group"
The pfizer results are dependent on the accuracy of the PCR test.
"The false-negative rate for SARS-CoV-2 RT-PCR testing is highly variable: highest within the first 5 days after exposure (up to 67%), and lowest on day 8 after exposure (21%)."
2. PCR tests do not show previous infection. Meaning that Pfizer PCR testing would miss all those people that had "flu-like symptoms" had covid, and recovered prior to getting tested. Did the pfizer study accounts for such likely scenarios?
Finally, why were there so many people with "Flu-like symptoms" in the pfizer study yet they did not have covid or the flu? Since we know that flu infections were at record lows during the last two years. Any explanation for this?
That's 94%, and it was true vs the original strain immediately after vaccination. Now many people are six months or more past their last shot and the current virus is two years of evolution improved vs the original. Vaccination reduces personal risk significantly but it's not going to control spread by itself.
