Scientific studies create headline news all the time, and for good reason. World-changing findings are happening in different fields of studies all the time. This has given rise to pop science, or a translation of scientific findings for the general public. While that fosters a greater understanding and interest in science, it can also lead to misunderstandings and bad science.
Bad science is a combination of poorly done research and scientific publication, as well as the translation of the findings for the wider public. This loose definition means that anyone can create bad science, from the researchers to the journalists. Even you can make bad science by misinterpreting a study and sharing that misinterpretation.
Sadly, bad science is often used to justify fake news or a political agenda. The past is littered with examples of bad science leading people astray. It is up to each and every one of us to combat the spread of bad science by knowing what to look for and being informed citizens. Here are the signs of bad science.
Untrustworthy Studies
While we want to believe that science is immune to bias or impeccably researched, reality isn’t always perfect. Sometimes, perfectly innocent flaws flavor a study. Experts tend to hold each other accountable, which is why academic publishing has safeguards in place. A few of these safeguards are pretty clear when you look at the source material of the study.
What are the Grounds for this Study?
If the study doesn’t source any of their claims or use disreputable sources, there’s a chance that the findings are disreputable as well.
Most reputable studies will back up their claims by linking to other studies that reinforce their claims. If the study doesn’t source any of their claims or use disreputable sources, there’s a chance that the findings are disreputable as well.
If a study is in a new field, there may not be a lot of sources to fall back on. In this case, replication becomes important. Replication is when another group of scientists test out the findings to see if they get the same results. If they do, the original study is verified.
Was It Reviewed?
Referees put their own reputations on the line to back up the study.
A peer review means the study was examined by peer experts, called referees, who are usually hired and vetted by a publishing academic journal, to verify the quality of the study. Referees put their own reputations on the line to back up the study.
Who’s the Publisher?
While a peer review is the academic norm, some publishers have chosen to buck these norms.
Where a study is published does matter. While a peer review is the academic norm, some publishers have chosen to buck these norms, for whatever reason. These are known as predatory publishers, since they’ll often charge large fees in a pay-to-play manner. These studies are rarely reviewed in the proper scientific rigor that is necessary.
Who Funded It?
If the authors try to hide any financial conflicts, this is much more concerning.
You may not want to trust research about the health risks of smoking if it was funded by the cigarette industry. This should only be one factor in your analysis. If the study is well researched, peer reviewed, and published in a reputable journal, the funding becomes less important. This is especially true if the researchers are upfront about who funded them. If the authors try to hide any financial conflicts, this is much more concerning.
The rigorous review process in scientific publishing is a primary reason we hold science in such high esteem today. They’re in place to catch bad science before it gets to the rest of us. It’s up to us to catch what slips through the cracks.
Flawed Analysis of Results
Unfortunately, good science can become bad when it reaches the public. A study’s data may be perfectly sound and verifiable, but if we misinterpret the findings or twist them to fit a bias, they become warped. If you’re reading an article online about a study, there are still signs you should watch for. One of the first signs that should tip you off to bad science is the article headline.
If the article is making bold, definitive claims like “scientists find cure for all cancer,” sirens should be going off for you. Science doesn’t really deal in definites and there’s no such thing as scientifically-proven. It’s why we don’t use that phrase on the Shop & Enroll blog and why gravity is still technically only a theory.
As you look through the article, keep an eye out for analysis of the study’s findings. One of the primary mistakes pop science makes is mistaking correlation and causation. Correlation is when two things happen together, while causation is when one thing causes another.
One of the primary mistakes pop science makes is mistaking correlation and causation.
If the subjects of the “scientists find cure for all cancer” study all ate two apples a day and didn’t get cancer over the course of the study, it’d be incorrect to believe that eating those apples was the cure, but merely a correlation. Correlation can imply causation, but until further study is completed, to say one causes the other is incorrect.
If you’re wary of a report, it usually helps to find the original study, which should be linked in the article you’re reading. Read the results and analysis that the researchers themselves found to see if the article correctly interpreted them. While you’re looking at the study, there are a few signs you can look for including sample size and type of subject. Sample size is how many subjects (humans, animals, etc.) were studied. If the sample size (usually denoted with an “n=”) is tiny, it’s probably not representative of the whole.
Analysis can also be flawed if an article makes broad claims when the subject groups are limited or very specified. For example, if the researchers studied a group of 30 middle-aged men from China, it may not be a statistically significant study because the sample size is so small and so specific. In this case, it would be incorrect to claim that “scientists find cure for all cancer” based on the findings of this study.
An Example of Bad Science
One case of questionable science is also one that’s become very controversial in popular culture. While everyone has the right to make their own health decisions, one study in particular left decision-makers with incomplete information regarding the claimed side effects of vaccination.
A 1998 Lancet study by Andrew Wakefield et al famously found a causal link between the measles, mumps, and rubella (MMR) vaccination and autism, but shortly after, several studies were published that actually refuted the findings of the Wakefield study. This was followed by a retraction of the interpretation of the study by 10 of the 12 authors.
The journal Lancet would also admit that Wakefield and company had failed to disclose financial conflicts and completely retracted the study. The study’s story would end with an investigation finding that Wakefield had selectively picked results and even falsified findings to fit a desired outcome, committing fraud in the process.
Unfortunately, the Wakefield study hit all four of our ‘bad science’ categories.
We picked this study not because of the case series’ subject matter, but because of the bad science it used to get to that point. Unfortunately, it hit all four of our “bad science” categories. It was an untrustworthy study due to the financial complication of Wakefield and company as well as evidence of bias ruining the findings.
The Wakefield study was also guilty of too small a sample size and confusing correlation with causation, which led to it being retracted entirely by Lancet and all but two of the authors.
Finally, the flawed analysis led to poor reporting on the study. “Vaccinations cause autism” was a headline claiming scientific certainty that was impossible based on a limited sample size. The report led to a significant drop in vaccinations in the United Kingdom and United States, showing how it captured the public imagination. If anything, this study shows the power that bad science can hold over us.
● ● ●
The authority that scientists and doctors wield, especially in their expert fields, is well-founded and well-deserved. This doesn’t mean that every study produced is perfect or without error and should be believed inherently. We’re all human, and we all have a part to play in the fight against bad science. Now you know what to look for.
Further Reading
Gizmodo — The Worst-Reported Science Stories of 2017
LiveSCIENCE — Oops! 5 Retracted Science Studies
Vox — 15 ways to tell if that science news story is hogwash
