Thinking in bets

Two researchers went looking for an answer. Frederick Mosteller and Cleo Youtz gathered up twenty studies. People had been asked to put a number on words. How likely is likely? How often is frequent? Some words came back tight. Others came back wildly apart. The worst offender was possible.

You carry two things inside any belief. One is what you think is true. The other is how sure you are about it. They travel together, so they feel like a single thing. They are not. They can come apart. And, it turns out, they can be measured apart.

“Two aspects of knowledge are what one believes to be true and how confident one is in that belief.”— Baruch Fischhoff, Paul Slovic and Sarah Lichtenstein, Journal of Experimental Psychology: Human Perception and Performance, 1977

So swap pretty sure for a number. Say seventy percent instead. Nothing about your belief changed. But now you have made a promise. The promise is not about this one guess. It covers every single time you say seventy.

Calibrated means your numbers mean what they say. Take every time you said seventy percent. Put them all in one pile. About seventy in a hundred should have turned out true. Not this one guess. The pile.

One group has done this in public for sixty years. The US National Weather Service began this in 1965. Its forecasters gave a chance of rain, not a yes or no. Millions of those forecasts went out in the first ten years alone. Every one of them had a day when the answer arrived.

Two researchers wrote the test out plainly. Their names were Allan Murphy and Robert Winkler. Say a forecaster used thirty percent fifty times. Perfect would be rain on exactly fifteen of them. Fewer or more, and the thirty was off. That is the entire idea. Count, then compare.

Murphy and Winkler then went and checked a real record. Chicago forecasters, July 1972 to June 1976. That is 17,514 rain forecasts with known outcomes. Forecasts go out three or four times a day. So these are forecasts, not days. Thirty percent was the call on 1,574 of them.

Weather is not special because rain is easy to guess. It is special because the answer always arrives. Every forecast gets marked within a day. Nobody can quietly forget the ones that missed. Do that for years, and the numbers get honest.

Here is where most people go wrong. You said seventy percent, and it did not happen. Were you wrong? There is no answer to that question. A seventy that fails is what seventy predicts. It should fail three times in ten.

You cannot score one number. You can score a hundred of them. Sort every guess by what you said at the time. Then count what actually happened inside each group. The counting is the whole method.

A second question shows up here, and it is easy to miss. Do your numbers mean what they say? And do your numbers tell anyone anything at all? Those are different questions. Scorers keep them in separate columns.

There is a cheap way to be perfectly calibrated. It takes no skill and no knowledge. See if you can spot the catch.

Forecast scorers call the missing part resolution. Resolution asks whether you tell the days apart. Did you push some numbers up and others down? And were you right to? A forecaster who never moves has none of it.

Calibration keeps you honest. Resolution makes you worth asking. You can hold one without the other. The best forecasters do both at once. They push hard, and they still land on their numbers.

In 2013, researchers ran this on 118 doctors. Each doctor read four written cases. Two were built to be easy. Two were built to be hard. They named a diagnosis for each one. Then they rated their own confidence, from zero to ten.

On the easy cases the doctors got 55.3 percent right. Then the hard cases arrived.

The same study watched what the doctors asked for. When confidence went up, requests for extra tests went down. Whether they were actually right showed no such pattern. The feeling was steering what they asked for. Being right was not.

Confidence is something you feel now. Accuracy is something somebody counts later. They can rise together, or they can come apart. From the inside, both cases feel the same. Only the record tells you which one happened.

People have been measuring this since the 1970s. In 1977, three researchers published five studies on it. Baruch Fischhoff, Paul Slovic and Sarah Lichtenstein. They asked questions with two possible answers. People picked one, then stated the odds they were right.

Odds were the clever part of the design. Percents run out of room at the top. Odds do not. Two to one is a mild lean. A hundred to one means you expect one miss in 101. A million to one means you would bet the house.

So people gave their odds, question after question. Then the researchers counted the answers.

Accuracy did climb higher up the ladder. It never came close to the promise. At 1,000 to 1, people were right 81 percent of the time. At 10,000 to 1, 87 percent. At a million to one or more, 90 percent. For those, honest odds were about 9 to 1.

This was not a rare flourish at the top of the scale. Of 6,996 odds judgments, 3,560 sat above 50 to 1. That is 51 percent of them. Almost a quarter went above 1,000 to 1. Extreme certainty was the normal setting.

The researchers wondered if people just misread the scale. So they ran the whole thing again with a lecture first. Twenty minutes on probability, odds and calibration. People were told plainly to aim for being calibrated.

In a later study, thirteen people played a betting game. The bets were built out of their own odds. Look at the answers they had rated 50 to 1 or better. They missed 46 of those 387 answers. That is one miss in eight. They had promised one in fifty-one.

You might hope this comes from a handful of odd people. The researchers checked exactly that. Most people had several cases of extreme overconfidence. Two people were far worse than the rest. The researchers pulled those two out, and nothing changed.

Now hold the study at the right size. It is from 1977, with paid volunteers in Oregon. The questions were trivia, not their own lives. Nothing here was riding on the answer. It shows the effect is real and easy to produce. It does not measure you.

One word gets used for three different habits. In 2008, Don Moore and Paul Healy pulled them apart. Each one is measured a different way. And they do not always show up together.

Overprecision is the habit that will not budge. A common test asks for a range instead of an answer. How long is the Nile? Give a range you are ninety percent sure holds the answer. Wide ranges are allowed. Nobody is timing you.

It is tempting to finish here with a slogan. You are overconfident. Be humbler. That is too tidy, and it is not what the data says. Moore and Healy found the flip on easy tasks. There, people often score better than they guessed they would.

In 2011 something unusual happened. A research arm of the US intelligence services ran a contest. University teams recruited forecasters from the public. The forecasters answered questions about world events. Over two years, 199 questions closed with a known answer.

Scoring starts from one plain idea. You said 0.7. The thing happened, which counts as 1. The gap between them is 0.3. Square that gap, and this guess costs you 0.09. Do it for every guess you ever made.

Squaring is the part that makes the score honest. A small miss stays small. A loud, confident miss gets punished hard.

Average those squared gaps across all your guesses. That average is your score. Lower is better, and zero is perfect. Glenn Brier proposed it for weather forecasts in 1950. Tournaments still score people with it today.

One detail matters before the tournament numbers arrive. That tournament added the gap on both sides. It scored what happened and what did not. That doubles every score. So their worst is 2, not 1. And always saying fifty lands on 0.50, not 0.25.

A useful property hides inside the squaring. It quietly removes any reason to shade your number.

Now the two ideas from earlier come back. The score can be pulled apart into three pieces. One is calibration. One is resolution. The third is just how hard the questions were. That piece is about the questions, not about you. Two people with the same score can fail in opposite ways.

The tournament also tested training. One group read a short module before starting. It told them to find a comparison class first. To average several estimates instead of trusting one. And to watch for known traps like overconfidence. The whole thing took about forty-five minutes.

Then the researchers watched what happened next. They tracked the scores through both tournament years. The trained group read that module once, at the start. They could go back to it. That was all.

A later paper followed four years of these tournaments. The same training still ran under an hour. Every year, the trained group's error score came out lower. The paper puts the gap at about 6 to 11 percent.

Year one's best forecasters were put on their own teams. The researchers called them superforecasters. In year two they averaged 0.25 in a question's first week. By the last week they averaged 0.07. Untrained people working alone averaged 0.46, then 0.26.

The paper looked at what the top group did differently. Much of their edge came from resolution. They moved their numbers further, and they were right to. They were better calibrated too. They also updated far more often. About 7.8 forecasts per question, against 1.4 working alone.

None of this works on a vague worry. You need a question that will answer itself one day. That takes three things: a date, a line, and a source. The date says when. The line says how much. The source says who gets to decide.

The number has to go down before the answer arrives. This sounds fussy. It is the whole thing. Afterwards, memory quietly edits how sure you felt. Cognitive biases calls that hindsight bias. A written number is the one version that survives it.

The practice itself is small. It fits inside a notes app. Only the order really matters.

After a few months you have a pile. Group every guess by the number you gave it. All your sixties together, all your nineties together. Count how many in each group came true. That is exactly what Murphy and Winkler did with Chicago.

Say your nineties come true seventy percent of the time. That is not a failure. That is a measurement. It tells you that your ninety means seventy. Next time, the same feeling gets written down as seventy.

A record does not tell you a choice was good. It tells you whether your numbers mean what they say. Those are different things. A well calibrated person still loses plenty of bets. They just lose about as often as they said they would.

Here is what all of that looks like on an ordinary Tuesday. Nothing about it needs a spreadsheet.

This does not make you right more often, by itself. It makes your numbers mean something. That is a smaller claim than it sounds. It is also a real one. It turns a feeling into something that can be checked.

Weather forecasters did not get calibrated by trying harder. They got there because the answer came back every day. Almost nobody else is scored like that. A written record is how people build that for themselves. The poker player Annie Duke wrote a book on it. She called it Thinking in Bets.