How do Superforecasters make Predictions?

From Philip Tetlock’s “Superforecasting”

Here is the Method Superforecasters Use to make Predictions. (Wording is from me, not book)

1. They Break Down the Question into Sub Questions. “Decompose the problem into its knowable and unknowable parts.” (p. 27)
   
   
   
2. They Get the Outside View. (The Outside View is the Base Rate. It is how common something is within a broader class). They ignore the particular details about a case.
   
   
   
3. They Get the Inside View. (The Inside View is the specific details about a case.) They gather as much data as possible from as many quality sources as possible.
   
   
   
4. They Start with the Outside View and Adjust it with the Inside View. They adjust the Outside View up or down, depending on how strong is the Validity of the Inside View. “So you have an outside view and an inside view. Now they have to be merged, just as your brain merges the different perspectives of your two eyeballs into a single vision.” (p. 121)
   
   
5. They Make A Forecast. For example, they will say I believe there is a 20% chance there will be a clash between China and Vietnam in the next 12 months.
   
   
6. They Constantly Update Their Forecast. They constantly look for updated information and adjust their forecasts.
   
   
7. They Share Their Forecasts With Other Superforecasters. They discuss and get input from other Superforecasters. They are openminded to hear information that may disconfirm their opinion.
   
   
   
   
   
   
   

Here is the Method from the book’s wording

• “Unpack the question into components. Distinguish as sharply as you can between the known and unknown and leave no assumptions unscrutinized.”
  
  
• “Adopt the outside view and put the problem into a comparative perspective that downplays its uniqueness and treats it as a special case of a wider class of phenomena.”
  
  
• “You may wonder why the outside view should come first. After all, you could dive into the inside view and draw conclusions, then turn to the outside view. Wouldn’t that work as well? Unfortunately, no, it probably wouldn’t. The reason is a basic psychological concept called anchoring.” (p. 120)
  
  
• “Then adopt the inside view that plays up the uniqueness of the problem.”
  
  
• “Also explore the similarities and differences between your views and those of others—and pay special attention to prediction markets and other methods of extracting wisdom from crowds.”
  
  
• “Synthesize all these different views into a single vision as acute as that of a dragonfly.”
  
  
• “Finally, express your judgment as precisely as you can, using a finely grained scale of probability.” (p. 153)
  
  
  
  

Examples of Forecasting Questions.

Example 1:

“The Renzettis live in a small house at 84 Chestnut Avenue. Frank Renzetti is forty-four and works as a bookkeeper for a moving company. Mary Renzetti is thirty-five and works part-time at a day care. They have one child, Tommy, who is five. Frank’s widowed mother, Camila, also lives with the family.” (p. 117)

How likely it is that the Renzettis have a pet?

Most people will look at the Inside View first. They look at details of the family and give a prediction based on their intuition. They might think:

“Renzetti is an Italian name,” someone might think. “So are ‘Frank’ and ‘Camila.’ That may mean Frank grew up with lots of brothers and sisters, but he’s only got one child. He probably wants to have a big family but he can’t afford it. So it would make sense that he compensated a little by getting a pet.”

Someone else might think, “People get pets for kids and the Renzettis only have one child, and Tommy isn’t old enough to take care of a pet. So it seems unlikely.” This sort of storytelling can be very compelling, particularly when the available details are much richer than what I’ve provided here.” (p. 117)

This is a mistake! You should not start with the Inside View (detailed information about the Renzettis). You should start with the Outside View! You should ignore all the details about the family. Once you get the Outside View, you can start to adjust the Outside View with the Inside View.

• What is the Outside View for families with pets? If you google it, it shows 62% of American households own pets. So the Outside View (base rate) is 62%.
  
  
• Now gather all details about the Renzettis. This is called the Inside View.
  
  
• Now adjust the Outside View with the information from the Inside View. You can adjust the Outside View up or down depending on the validity of the inside view.
  
  
• Note: It’s possible to find more than just one Outside View. Perhaps you found the Outside View for single family homes. The Outside View for single family homes is 73%. Since 84 Chestnut Hill Ave is a single family home, you should start with 73% and adjust it up or down depending on the Validity of the Inside View.
  
  
• “It’s natural to be drawn to the inside view. It’s usually concrete and filled with engaging detail we can use to craft a story about what’s going on. The outside view is typically abstract, bare, and doesn’t lend itself so readily to storytelling. So even smart, accomplished people routinely fail to consider the outside view.” (p. 118)

Example 2:

Is there going to be an armed clash between China and Vietnam over border disputes in the next 12 months?

• First get the Outside View (base rate). Ignore all of the current political news about the relationship between China and Vietnam. The base rate shows on average there is one hostile clash between China and Vietnam every 5 years. So the Outside View shows there is a 20% chance of a clash in 1 year.
  

• Now adjust the Outside View with the Inside view. Gather more information by looking at the current political relationship between China and Vietnam. You can adjust the 20% up or down.
  
  
  
  

Example 3:

Terrorists had just murdered eleven people at the Paris satirical magazine Charlie Hebdo. (early 2015)

“Will there be an attack carried out by Islamist militants in France, the UK, Germany, the Netherlands, Denmark, Spain, Portugal, or Italy between January 21 and March 31, 2015?” (p. 121)

• Get the Outside View. The media is going to be filled with stories and information about that attack, but ignore that information. Simply count all the terrorist attacks that happened in the specified countries, and then divide that number by the number of years it happened. By checking Wikipedia, you can see there was 6 attacks in the past 5 years. That averages to 1.2 attack a year. The Outside View is 1.2 attacks a year.
  
  
• Now get the Inside View.
  
  “In the previous several years, the Islamic State of Iraq and Syria (ISIS) movement had surged to prominence. Hundreds of European Muslims had enlisted. And ISIS had repeatedly threatened Europe with terror attacks.” With this new information, you might discard the first two years of information. If you remove the first two years information, the Outside View is now 1.5 attacks a year.
  

• Let’s say you gather more information and you think the attacks will be slightly more in the future. So now you update 1.5 attacks a year to 1.8 attacks a year. Your adjusted view is now 1.8 attacks a year.
  
  
• Between Jan. 21- March 31st. there are 69 days. 69 days out of 365 days is 19% of the year.
  
  
• 19% X 1.8 = 34%. You now predict there is a 34% chance there will be an attack from Jan. 21- March 31st.
  
  
  

Example 4: (Real example from the tournament)

“On October 12, 2004, Yasser Arafat, the seventy-five-year-old leader of the Palestine Liberation Organization, became severely ill with vomiting and abdominal pain. Over the next three weeks, his condition worsened. On October 29, he was flown to a hospital in France. He fell into a coma. Decades earlier, before adopting the role of statesman, Arafat had directed bombings and shootings and survived many Israeli attempts on his life, but on November 11, 2004, the man who was once a seemingly indestructible enemy of Israel was pronounced dead. What killed him was uncertain. But even before he died there was speculation that he had been poisoned.”

“In July 2012 researchers at Switzerland’s Lausanne University Institute of Radiation Physics announced that they had tested some of Arafat’s belongings and discovered unnaturally high levels of polonium-210. That was ominous. Polonium-210 is a radioactive element that can be deadly if ingested. In 2006 Alexander Litvinenko—a former Russian spy living in London and a prominent critic of Vladimir Putin—was murdered with polonium-210. That August, Arafat’s widow gave permission for his body to be exhumed and tested by two separate agencies, in Switzerland and France.”

“Will either the French or Swiss inquiries find elevated levels of polonium in the remains of Yasser Arafat’s body?” (p. 114)

How did the Superforecaster answer this?

• Get the Outside View. – How often is poison detected in cases like this?
  
  
• “what is the outside view in the Arafat-polonium question? That’s tough. It’s not as if dead Middle Eastern leaders are routinely exhumed to investigate suspicions of poisoning—so there’s no way we’re going to be able to do a quick Google search and find that poison is detected in 73% of cases like this. But that doesn’t mean we should skip the outside view and go straight to the inside.”
  
  
• “Let’s think about this Fermi-style. Here we have a famous person who is dead. Major investigative bodies think there is enough reason for suspicion that they are exhuming the body. Under those circumstances, how often would the investigation turn up evidence of poisoning? I don’t know and there is no way to find out. But I do know there is at least a prima facie case that persuades courts and medical investigators that this is worth looking into. It has to be considerably above zero. So let’s say it’s at least 20%. But the probability can’t be 100% because if it were that clear and certain the evidence would have been uncovered before burial. So let’s say the probability cannot be higher than 80%. That’s a big range. The midpoint is 50%. So that outside view can serve as our starting point.” (p. 119)
  
  
• Now get the Inside View (detailed information about the case)-
  
  “Again, Fermi-ization is key. When Bill Flack Fermi-ized the Arafat-polonium question, he realized there were several pathways to a “yes” answer: Israel could have poisoned Arafat; Arafat’s Palestinian enemies could have poisoned him; or Arafat’s remains could have been contaminated after his death to make it look like a poisoning. Hypotheses like these are the ideal framework for investigating the inside view. Start with the first hypothesis: Israel poisoned Yasser Arafat with polonium. What would it take for that to be true?
  
  
  1. Israel had, or could obtain, polonium.
  2. Israel wanted Arafat dead badly enough to take a big risk.
  3. Israel had the ability to poison Arafat with polonium.
  
  Each of these elements could then be researched—looking for evidence pro and con—to get a sense of how likely they are to be true, and therefore how likely the hypothesis is to be true. Then it’s on to the next hypothesis. And the next.” (p. 121)
  
  
• “After Bill Flack did all his difficult initial work and concluded there was a 60% chance that polonium would be detected in Yasser Arafat’s remains, he could raise or lower his forecast as often as he liked, for any reason. So he followed the news closely and updated his forecast whenever he saw good reason to do so. This is obviously important. A forecast that is updated to reflect the latest available information is likely to be closer to the truth than a forecast that isn’t so informed.” (p. 153)
  
  
• The Swiss team’s delay suggested it had detected polonium and was now testing to rule out lead as its source. But that was only one possible explanation, so Bill cautiously raised his forecast to 65% yes. That’s smart updating. Bill spotted subtly diagnostic information and moved the forecast in the right direction before everyone else—as the Swiss team did, in fact, find polonium in Arafat’s remains.” (p. 157)
  
  
  
  

Other Good Quotes About Making Predictions:

“This sounds like detective work because it is—or to be precise, it is detective work as real investigators do it, not the detectives on TV shows. It’s methodical, slow, and demanding. But it works far better than wandering aimlessly in a forest of information.” (p.121)

“It’s natural to be drawn to the inside view. It’s usually concrete and filled with engaging detail we can use to craft a story about what’s going on. The outside view is typically abstract, bare, and doesn’t lend itself so readily to storytelling. So even smart, accomplished people routinely fail to consider the outside view.” (p. 118)

“For one thing, superforecasters’ initial forecasts were at least 50% more accurate than those of regular forecasters. Even if the tournament had asked for only one forecast, and did not permit updating, superforecasters would have won decisively.” (p. 155)

“Researchers have found that merely asking people to assume their initial judgment is wrong, to seriously consider why that might be, and then make another judgment, produces a second estimate which, when combined with the first, improves accuracy almost as much as getting a second estimate from another person. The same effect was produced simply by letting several weeks pass before asking people to make a second estimate. This approach, built on the “wisdom of the crowd” concept, has been called “the crowd within.”” (p.123)

“The billionaire financier George Soros exemplifies it. A key part of his success, he has often said, is his mental habit of stepping back from himself so he can judge his own thinking and offer a different perspective—to himself.” (p. 123)

“Will the South African government grant the Dalai Lama a visa within six months?” The naive forecaster will go looking for evidence that suggests the Dalai Lama will get his visa while neglecting to look for evidence that suggests he won’t. The more sophisticated forecaster knows about confirmation bias and will seek out evidence that cuts both ways.”

You should ask “Will the South African government deny the Dalai Lama for 6 months?” (p. 123)

A man’s height is 6 feet. How tall do you predict the son to be?

• What is the base rate? The average height of man is 5 foot 8 inches. So the base rate is 5 foot 8 inch.
  
  
• What is the correlation for height between a man and son? The height Coefficient is 0.5.
  
  
• Start with the Outside View (5 foot 8 inches), and adjust with the inside view (6 foot). Since the height coefficient is 0.5, the answer is 5 foot 10 inches.