Loading
Someone says a study proved it. Maybe a friend. Maybe a headline you scrolled past. You will not read that study. Almost nobody does.
Many papers sit behind a paywall. They are written for other scientists. Even the free ones are hard going.
So you need something else. You need better questions. Good questions work from outside the paper. That is what this module hands you.
"Studies show" tells you a paper exists. It does not tell you the paper is good. A weak study and a strong one look alike in a headline.
People argue about whether evidence exists. That is usually the wrong fight. Almost every side has some. The real question is how much weight it holds.
Two claims about the same pill. Which tells you more?
You cannot judge the science. You can judge the setup. The setup is usually described in plain words. It is often the only part you can read.
Every result is a comparison. Find the other group.
Those five are the whole module. The rest is what each one catches, and why. None of them need a science degree.
These questions will not tell you who is right. They will not make you a scientist. They do something smaller and more useful.
They tell you how hard to lean on a claim. Sometimes the answer is: not very hard at all. That is worth knowing before you act on it.
Not quite, and the gap matters. You can spot a badly built study. That is not the same as a wrong answer. A careful study can land wrong by bad luck. A sloppy one can stumble onto the truth. These questions change how much weight you give a claim. They do not settle it.
Every result is a comparison. Always. "The pill worked" means one group did better than another. If nobody names the other group, the sentence is unfinished.
That group is the ruler you measure against. Ask the question out loud on the next claim you hear. A lot of them fall apart right there.
A cold ends. A sore back eases. A bad week passes. Try something on your worst day and you will feel better. You would have felt better anyway.
The new thing takes the credit. This is why a control group is not paperwork. Without one, you cannot separate the treatment from time.
Time heals a great deal and gets none of the credit. So does hope. So does simply being watched by a doctor.
In 1976 a New Jersey prison tried something new. Men serving life sentences shouted at young offenders. The idea was to scare them straight.
A film about it won an Academy Award. Other states copied the programme quickly. The programme claimed most of the young people stayed out of trouble.
You could see it working with your own eyes. They came out shaken and sorry. That is the part everybody trusted.
Researchers ran it as a proper trial. Young people were split into two groups. One group went to the prison. One group did not.
Nine trials tested prison visits like this one. They ran in the United States between 1967 and 1992. All of them followed both groups afterwards.
What did the trials find?
In 2013 a team gathered all nine trials. They judged them together and wrote one conclusion. Here it is in their own words.
“We conclude that programs such as 'Scared Straight' increase delinquency relative to doing nothing at all to similar youths.”— Petrosino and colleagues, Cochrane Database of Systematic Reviews, 2013
Everyone at the prison saw a real effect. They were not lying and they were not fools. They were missing one group.
Both halves of that are true. They really were shaken. Many really did stay out of trouble. The part nobody could see was the young people who never went. Those ones stayed out of trouble more often. Without that second group, a real change and no change look identical.
There is a second half to the question. Compared with what, exactly? Beating nothing at all is easy.
Beating the best treatment we already have is hard. Both get written up the same way. Both become "it worked."
Same result. Very different news.
Two of these five have no other group at all. The wording hides it well. Read each one for the missing half.
Which of these has a real comparison group?
Tap an item, then a bucket
A comparison only works if the groups begin the same. Same ages. Same health. Same in ways nobody measured. If they start different, the ending tells you nothing.
You already know that correlation is not causation. This is the question one step below it. How did these two groups come to exist?
Somebody assigned them. Or they sorted themselves. Those two studies are read very differently. Only one of them can show a cause.
Say people join a running club. A year later they are healthier than their neighbours. The running may have done it.
Or the kind of person who joins was healthier already. Afterwards you cannot pull those two apart. The choice walks into the results with the person.
People who take vitamins tend to sleep more. They tend to smoke less and see doctors sooner. The pill gets credit for all of it.
In 1981 four researchers pointed at something odd. People who ate more of one orange pigment got less cancer. The pigment is beta-carotene.
It is in carrots, spinach and sweet potato. The researchers asked for the obvious next step. Somebody should test it properly, with pills.
A large trial ran in Finland. About 29,000 men who smoked took part. A coin flip decided who got beta-carotene pills.
They were followed for five to eight years. This was the test the 1981 paper had asked for. Everybody expected the pills to help.
What did the pills do?
A second trial was already running in the United States. It gave beta-carotene and vitamin A to smokers. It also included workers who had breathed asbestos.
About 18,000 people were in it. It was stopped 21 months early. The group taking the pills was doing worse.
More lung cancer. More deaths. Stopping a trial early is rare. It is never done lightly.
The eating data was not wrong. People who ate those foods really did get less cancer. But nobody had assigned them to eat that way.
They chose it. Eating spinach arrives bundled with a whole life. More sleep, more money, more walking, fewer cigarettes.
The pill was the pigment on its own. The rest of the bundle did not come with it. Only a trial could have shown that.
Random assignment sounds fussy. It is the strongest move a study can make. A coin flip does not know who is rich or careful.
So the two groups come out alike. Alike even in ways nobody thought to measure. That last part is what nothing else can do.
There is a second trick, almost as important. The people in the study are not told their group. Where possible, the people measuring are not told either.
Someone who knows tends to find what they hoped for. Not by cheating. By reading a borderline case kindly, without noticing.
You cannot assign people to smoke for thirty years. That study would be monstrous. So some questions can only be studied by watching.
That work still counts. Smoking and lung cancer was settled that way. So was most of what we know about diet.
It just has to be read differently. Watching tells you where to look. Assigning tells you what happens when you change one thing.
Here are five study shapes. Ask who decided which group each person landed in. The wording will not tell you outright.
Did the researchers set the groups, or did people sort themselves?
Tap an item, then a bucket
There is a twin question, and it is easy to miss. Not just compared with what. Compared with whom. The two groups can differ before anyone treats them.
A big hospital with new scanners reports worse survival. Why might that be?
A new result gets the most attention on day one. That is when it has earned the least trust. Nobody has tried it again yet.
In 2015 a large group of researchers ran a test. They took 100 published psychology studies. They ran every one of them again.
In the first 100 reports, the effect showed up 97 times. How many times on the second run?
Other teams tried the same exercise elsewhere. One group repeated 21 social science experiments. They came from two of the most famous journals.
The result came back for 13 of them. Again at about half the size. Eight of those famous findings did not come back.
Another group spent eight years on cancer laboratory work. Of the repeats they could measure, 92 in 100 came out smaller. The rest came out bigger.
There is a dull reason, and it is not dishonesty. A result gets noticed partly because it came out big. Some of that size was luck.
Luck does not turn up twice to order. So the second look shrinks. Expect that, and you will not be shocked by it.
If you have read Statistics for everyday life, you met this drift. Here the point is narrower. A shrinking repeat is normal, not a scandal.
Running someone else's study again is called replication. It is slow and it costs money. No journal is thrilled to print it.
So it happens far less than you would guess. That is why "one study found" is so common a sentence. Often one study really is all there is.
Sometimes a team hunts down every study on a question. Strong ones and weak ones. They judge each one, then combine the results.
A good review beats any single study inside it. If you can read only one thing, read one of these. Cochrane is the best known name that makes them.
Researchers read 462 press releases from British universities. About a third turned a link into a cause. Then they read the news written from those releases.
Where the release stretched a cause, 81 stories in 100 stretched it. Where it did not, only 18 in 100 did. Most of the exaggeration was there before any reporter saw it.
A headline says a new study links coffee to a longer life. What is the most useful next question?
Picture a drawer. Every study that found nothing goes in it. Nobody publishes it. Nobody reads it. It just sits there.
A psychologist named this in 1979. Almost nobody hides results on purpose. A researcher who found nothing simply has a dull paper.
Journals want new and surprising. A dull paper goes on a pile. Nobody lied. The record still comes out bent.
A drug company must tell the American regulator about its trials. It has to, in order to get a drug approved. So somebody could compare two lists.
Seventy-four antidepressant trials were checked. The regulator judged 38 of them positive. Of the 36 that were not, 22 never appeared anywhere.
Then they asked the simple question. If you only read the journals, what would you think? The two answers are not close.
“According to the published literature, it appeared that 94% of the trials conducted were positive. By contrast, the FDA analysis showed that 51% were positive.”— Turner and colleagues, New England Journal of Medicine, 2008
Same drugs. Same trials. Two completely different pictures. Most of the gap came from trials that never appeared. Some came from papers that reported a failure as a win.
That sentence sounds heavy. It is lighter than it seems. It says nothing about how many studies were run.
It counts for less than it sounds. The sentence only says such studies exist. It says nothing about how many were run in total. If ten were run and two came out well, those two are findable. The sentence stays true. It stays true in exactly the case where the drawer is fullest.
Nobody in this chain does anything wrong. That is what makes it so hard to stop. Put the steps in the order they happen.
Put these in the order they usually happen.
Put these in order
There is a fix, and it is almost annoyingly simple. Before you start, you write down your plan. You say what you will measure and what counts.
Then you post it in public with a date on it. That is all it is. A plan, in public, before the numbers exist.
Once the plan is public, silence becomes visible. Anyone can see a trial that started and never reported. The drawer has a window in it now.
It exposes a quieter move too. You cannot quietly decide what counted after seeing the numbers. The goalposts were written down before the game.
It does not ask anyone to be more honest. It makes the switch visible to anyone who checks. Whether people check is a separate problem.
Asking nicely did not work. Somebody had to make it a condition. In 2004 the main medical journal editors did.
“require registration of clinical trials in a public trials registry at or before the time of first patient consent for enrollment as a condition of consideration for publication”— International Committee of Medical Journal Editors, Recommendations (current wording)
That rule still stands today, in those words. In plain terms: register it, or we will not print it. It covered trials enrolling people after 1 July 2005.
Researchers looked at large heart trials. All were paid for by an American health institute. The public register opened in 2000, so they split there.
Before 2000, 17 of 30 trials found the treatment helped. After 2000, only 2 of 25 did. The same institute. The same kind of question.
Registration is the obvious suspect, and the authors said so. It is not proof. Other things changed in those years. The honest version: the drop is real, the cause is likely.
Some journals now go further than a public plan. They review the plan and promise to publish it. Whatever the results turn out to be.
The promise is made before anyone sees a number. This is called a registered report. Researchers compared the two kinds of paper.
About 96 in 100 ordinary psychology papers report the idea worked. What about registered reports?
A plan also pins down what you promised to measure. Without one, a team can measure twenty things. Then they report the one that came out well.
The paper reads as a clean success. One project checked 67 trials against their own plans. They came from five of the top medical journals.
On average each trial reported about five unplanned results. Corrections were requested for 58 of the trials. Fewer than half of those corrections were printed.
Trial plans sit in public registers. The biggest is ClinicalTrials.gov. The World Health Organization searches across many national ones.
You need no subscription and no science degree. Read what the trial said it would measure. Then read what the paper actually reported.
When those two lists do not match, that is worth knowing. It is the one check nobody expects a reader to do. It does not take long.
Look at the five fixes together. Four of them are one move: decide in advance, in public. The fifth is the only one that happens afterwards.
Match each problem with the thing built to catch it.
Tap one side, then its match
Studies cost money. Someone always pays. A company, a government, a charity, a university. The funder is usually named near the end.
Look for three short lines there. Funding. Conflicts of interest. Acknowledgements. They take a minute and almost nobody reads them.
A press release will not carry them. Neither will a news story. That is one reason a press release is not evidence.
A large review gathered 75 studies. They had all asked one question. Do company-funded drug and device trials favour the sponsor?
On average, yes they do. The reviewers then looked for the usual explanation. Were the company trials simply badly run?
They were not. Standard quality checks did not explain the gap. Whatever is happening is not sloppy method.
One case is easier to picture. Researchers gathered every review on one question. Do sugary drinks put weight on people?
They sorted the reviews by the authors' money. Some had taken money from the food industry. Some had not.
Twelve conclusions came from reviews with no industry money. Ten of those said yes. Six came from reviews that had taken it. Five called the evidence weak.
It would be easy to stop reading right there. Plenty of people do. That move is worth a second look.
That step is one too far. Company money is a reason to read harder, not to stop reading. Some company trials are the best-run in their field. A regulator watched every step. Thinking traps covers the move that goes wrong here. That move judges a claim by who made it. Funding tells you where to check. The comparison group, the plan, and whether other trials exist.
One last thing before you go. A headline of the usual kind, and the questions in order. There is no trick in it.
A friend sends you a headline about standing desks.
It says standing desks cut back pain in half. Two hundred office workers switched. Six months later they reported less pain.
"The evidence is weak" does not mean "this is wrong". It means nobody has looked hard enough yet. Those two get mixed up constantly.
Weak evidence for something cheap and safe may be enough. Weak evidence for something costly or risky is not. The stakes change what counts as enough.
How sure we are is one question. What to do about it is another. Keeping them apart is most of the skill.
One question, and then the summary. Think about which one tests the result itself. The other three describe the packaging.
Which of these would most increase your confidence in a claim?
You still cannot read the paper. That has not changed. What changed is what you can ask from outside it.
Compared with what. Assigned, or did they choose. Has anyone else found it. What went missing. Who paid.
None of them hand you a verdict. Together they tell you how hard to lean. That is the whole job.