From Daniel Kahneman “Thinking, fast and slow”
Beware of of Base Rate Neglect!
When people make a prediction of something, they underweight Base Rates or even neglect them. This is an illusion called Base Rate Neglect.
“There are two ideas to keep in mind about Bayesian reasoning and how we tend to mess it up. The first is that base rates matter, even in the presence of evidence about the case at hand. This is often not intuitively obvious.” (p. 154)
“The second is that intuitive impression of the diagnosticity of evidence are often exaggerated. The combination of WYSIATI and associative coherence tends to make us believe in the stories we spin for ourselves.” (p. 154)
“The essential keys to disciplined Bayesian reasoning can be simply summarized:” (p. 154)
- 1. “Anchor your judgment of the probability of an outcome on a plausible base rate.“
- 2. “Question the diagnosticity of your evidence.”
“Both ideas are straightforward. It came as a shock to me when I realized that I was never taught how to implement them, and that even now I find it unnatural to do so.” (p. 154)
Example:
- “A cab was involved in a hit-and-run accident at night. Two cab companies, the Green and the Blue, operate in the city. You are given the following data:
- 85% of the cabs in the city are Green and 15% are Blue.
- A witness identified the cab as Blue.
- The court tested the reliability of the witness under the circumstances that existed on the night of the accident and concluded that the witness correctly identified each one of the two colors 80% of the time and failed 20% of the time.
“What is the probability that the cab involved in the accident was Blue rather than Green?” (p. 166)
You probably guessed 80%. That’s the wrong answer!
“The correct answer is 41%. Most people ignore the base rate and simply rely on the witnesses accuracy. Remember- “The first is that base rates matter, even in the presence of evidence about the case at hand. This is often not intuitively obvious.”
Use Bayes Theorem to Solve the Problem.
- First you should get the base rate. In order to get the base rate, you should ignore any extra information about the witness. The Base Rate that the cab will be blue is 15%.
- Now you can use extra information to adjust the base rate up or down. The reliability of the witness is 80% correct and 20% incorrect. Use Bayes Theorem and input all of these probabilities. The probability of the cab being blue is 41%.
- “The two sources of information can be combined by Bayes’s rule. The correct answer is 41%. However, you can probably guess what people do when faced with this problem: they ignore the base rate and go with the witness. The most common answer is 80%.” (p. 166)
Example 1.
“The following is a personality sketch of Tom W written during Tom’s senior year in high school by a psychologist, on the basis of psychological tests of uncertain validity:
“Tom W is of high intelligence, although lacking in true creativity. He has a need for order and clarity, and for neat and tidy systems in which every detail finds its appropriate place. His writing is rather dull and mechanical, occasionally enlivened by somewhat corny puns and flashes of imagination of the sci-fi type. He has a strong drive for competence. He seems to have little feel and little sympathy for other people, and does not enjoy interacting with others. Self-centered, he nonetheless has a deep moral sense.” (p. 147)
“Now please take a sheet of paper and rank the nine fields of specialization listed below by how similar the description of Tom W is to the typical graduate student in each of the following fields. Use 1 for the most likely and 9 for the least likely.” (p. 147)
business administration
computer science
engineering
humanities and education
law
medicine
library science
physical and life sciences
social science and social work
You probably ranked computer science as Tom’s most likely major. It fits in with Tom’s description of “high intelligence,” “neat and tidy systems” and “corny puns and flashes of imagination of the sci-fi type”.
That’s the wrong answer! You made a prediction based on the description of Tom. Yes, the description of Tom most likely matches a Computer Science major. The problem is you must also factor in the prior probabilities of each major!
It is not likely that Tom is a computer science major because there are many more Social Science and Humanity majors than Computer Science majors. You must look the Base Rate (also known as Outside View) Probability first. Then you can adjust the Base Rate with the Inside View (the new information or probability).
“The correct answer to the Tom W puzzle is that you should stay very close to your prior beliefs, slightly reducing the initially high probabilities of well-populated fields (humanities and education; social science and social work) and slightly raising the low probabilities of rare specialties (library science, computer science). You are not exactly where you would be if you had known nothing at all about Tom W, but the little evidence you have is not trustworthy, so the base rates should dominate your estimates.” (p. 153)
“For example, if you believe that 3% of graduate students are enrolled in computer science (the base rate), and you also believe that the description of Tom W is 4 times more likely for a graduate student in that field than in other fields, then Bayes’s rule says you must believe that the probability that Tom W is a computer scientist is now 11%. If the base rate had been 80%, the new degree of belief would be 94.1%. And so on.” (p. 154)
Example 2.
“Julie is currently a senior in a state university. She read fluently when she was four years old. What is her grade point average (GPA)?” (p. 186)
Most people will say 3.7 or 3.8. They will think since she’s in the 90% in reading, she will be in the 90% in her GPA as a senior. This is wrong! You must think about correlation coefficient and adjust for Regression.
Correlation Coefficient estimate- .3
Intuitive guess- 3.7
Average GPA- 3
Updated prediction
3.7- 3= 0.7
.3 X .7 – .21
3+ .21= 3.21
3.21 is the updates prediction.
Example 3
There is a group of 100 men. 70 of these men are engineers, and 30 of these men are lawyers.
“Dick is a 30-year-old man. He is married with no children. A man of high ability and high motivation, he promises to be quite successful in his field. He is well liked by his colleagues.” (p. 421)
What is the probability that Dick is a lawyer?
You probably guessed 50-50. That is wrong. You ignored the Base Rate of Lawyers. 30 out of these 100 men are lawyers. That means the Base Rate of Lawyers is 30%. That means the probability is 30% that Dick is a lawyer. The description doesn’t give you more evidence to make you think Dick has a 50% chance of being a lawyer.
Example 4
“Steve is very shy and withdrawn, invariably helpful but with little interest in people or in the world of reality. A meek and tidy soul, he has a need for order and structure, and a passion for detail.” (p. 7)
Is Steve more likely to be a farmer or librarian?
You probably said librarian. This is wrong. The Base Rate shows there are probably more than 20 male farmers to 1 librarian.
“Because there are so many more farmers, it is almost certain that more “meek and tidy” souls will be found on tractors than at library information desks.” (p. 7)
Once again you must consider the prior probabilities before looking at the description. You must look at the Base Rate or “Outside View”
Example 5.
“You see a person reading The New York Times on the New York subway. Which of the following is a better bet about the reading stranger?” (p. 151)
She has a PhD.
She does not have a college degree.
You probably said Phd. That’s the wrong answer.
How Can You Get a More Accurate Prediction?
- Get Outside View or Base Rate. Ignore the detailed information.
What is the Base Rate of a random person having a PhD? (Let’s guess1%)
What is the Base Rate of a random person having no college degree? (Let’s guess 33%)
- Now gather evidence or information for the Inside View. The Inside View is she reads the New York Times.
- Let’s say say you think 50% of all people with a PhD reads the NY times. So the Inside View is 50%.
So you start with the the Outside View of 1% and you anchor it with the Inside View of 50%. You final prediction that the person having a PhD should be 0.5%. - Let’s say you think only 10% of people with no college degree reads the NY Times. So the Inside View is 10%.
So you start with the Outside View of 33% and you anchor it with the Inside View of 10%. Your final prediction of someone that doesn’t have a college degree should be 3.3%.
“Representativeness would tell you to bet on the PhD, but this is not necessarily wise. You should seriously consider the second alternative, because many more nongraduates than PhDs ride in New York subways.” (p. 151-152)
Example 6
- “if you must guess whether a woman who is described as “a shy poetry lover” studies Chinese literature or business administration, you should opt for the latter option. Even if every female student of Chinese literature is shy and loves poetry, it is almost certain that there are more bashful poetry lovers in the much larger population of business students.” (p. 152)
Example 7
There is classic experiment called the “Helping Experiment.”
A man gives a speech in front of 15 people. Suddenly the man has a seizure and pleads for help. 4 out of the 15 people quickly respond to help him. 11 out of 15 people didn’t help him.
“Most of us think of ourselves as decent people who would rush to help in such a situation, and we expect other decent people to do the same. The point of the experiment, of course, was to show that this expectation is wrong.” (p. 171)
The base rate of people helping was 27% (4/15)
A Psychology Professor explained the “Helping Experiment” with his students. The students understood the lesson and understood Base Rates.
The Professor showed the students a video. It showed an interview with two of the individuals who were involved in the “Helping Experiment”
“The interviews were short and bland. The interviewees appeared to be nice, normal, decent people. They described their hobbies, their spare-time activities, and their plans for the future, which were entirely conventional.” (p. 172)
After watching the video, the students were asked if they thought the two individuals would help the man who was having the seizure. The students said yes. But they should have said no! They knew the base rate of quickly helping the man was only 27%! It’s most likely the 2 individuals didn’t help! They ignored the base rate!
“This is a profoundly important conclusion. People who are taught surprising statistical facts about human behavior may be impressed to the point of telling their friends about what they have heard, but this does not mean that their understanding of the world has really changed.” (p. 174)
“There is a deep gap between our thinking about statistics and our thinking about individual cases. Statistical results with a causal interpretation have a stronger effect on our thinking than noncausal information. But even compelling causal statistics will not change long-held beliefs or beliefs rooted in personal experience. On the other hand, surprising individual cases have a powerful impact and are a more effective tool for teaching psychology because the incongruity must be resolved and embedded in a causal story.” (p.174)
Other Good Quotes About Base Rates:
- “This may be considered the single most important piece of advice regarding how to increase accuracy in forecasting through improved methods. Using such distributional information from other ventures similar to that being forecasted is called taking an “outside view” and is the cure to the planning fallacy.” (p. 251)
- “There is one thing you can do when you have doubts about the quality of the evidence: let your judgments of probability stay close to the base rate. Don’t expect this exercise of discipline to be easy—it requires a significant effort of self-monitoring and self-control.” (P. 153)
- “The prevalent tendency to underweight or ignore distributional information is perhaps the major source of error in forecasting. Planners should therefore make every effort to frame the forecasting problem so as to facilitate utilizing all the distributional information that is available.” (P. 251)
- “Statistical base rates are generally underweighted, and sometimes neglected altogether, when specific information about the case at hand is available.” (p. 168)
- Causal base rates are treated as information about the individual case and are easily combined with other case-specific information. (p. 168)
- “The second is that intuitive impressions of the diagnosticity of evidence are often exaggerated. The combination of WYSIATI and associative coherence tends to make us believe in the stories we spin for ourselves.”