From Daniel Kahneman “Thinking, fast and slow”
Don’t Let Stories Distort Your View of Reality!
Let’s say you are trying to predict something. If you read a story or description of something, you will overweight this description as evidence and underweight the Base Rate. You have been affected by the Representativeness Bias!
- “Insensitivity to prior probability of outcomes. One of the factors that have no effect on representativeness but should have a major effect on probability is the prior probability, or base rate frequency, of the outcomes.” (p. 420)
Example 1.
“The following is a personality sketch of Tom W written during Tom’s senior year in high school by a psychologist, on the basis of psychological tests of uncertain validity:
“Tom W is of high intelligence, although lacking in true creativity. He has a need for order and clarity, and for neat and tidy systems in which every detail finds its appropriate place. His writing is rather dull and mechanical, occasionally enlivened by somewhat corny puns and flashes of imagination of the sci-fi type. He has a strong drive for competence. He seems to have little feel and little sympathy for other people, and does not enjoy interacting with others. Self-centered, he nonetheless has a deep moral sense.” (p. 147)
Now “rank the fields of specialization in order of the likelihood that Tom W is now a graduate student in each of these fields.”
business administration
computer science
engineering
humanities and education
law
medicine
library science
physical and life sciences
social science and social work
You probably ranked computer science as Tom’s most likely major. It fits in with Tom’s description of “high intelligence,” “neat and tidy systems” and “corny puns and flashes of imagination of the sci-fi type”.
That’s the wrong answer! You made a prediction based on the description of Tom. Yes, the description of Tom most likely matches a Computer Science major. The problem is you must also factor in the prior probabilities of each major!
It is not likely that Tom is a computer science major because there are many more Social Science and Humanity majors than Computer Science majors. You must look the Outside View Probability (Base Rate) first. Then you can adjust the Outside View with the Inside View (the new information or probability).
Example 2.
“Steve is very shy and withdrawn, invariably helpful but with little interest in people or in the world of reality. A meek and tidy soul, he has a need for order and structure, and a passion for detail.” (p. 7)
Is Steve more likely to be a farmer or librarian?
You probably said librarian. This is wrong. The Base Rate shows there are probably more than 20 male farmers to 1 librarian.
“Because there are so many more farmers, it is almost certain that more “meek and tidy” souls will be found on tractors than at library information desks.” (p. 7)
Once again you must consider the prior probabilities before looking at the description. You must look at the “Outside View” or Base Rate.
Example 3.
“Julie is currently a senior in a state university. She read fluently when she was four years old. What is her grade point average (GPA)?” (p. 186)
Most people will say 3.7 or 3.8. They will think since she’s in the 90% in reading, she will be in the 90% in her GPA as a senior. This is wrong! You must think about correlation coefficient and adjust for Regression.
Correlation Coefficient estimate- .3
Intuitive guess- 3.7
Average GPA- 3
Updated prediction
3.7- 3= 0.7
.3 X .7 – .21
3+ .21= 3.21
3.21 is the updates prediction.
Example 4
There is a group of 100 men. 70 of these men are engineers, and 30 of these men are lawyers.
“Dick is a 30-year-old man. He is married with no children. A man of high ability and high motivation, he promises to be quite successful in his field. He is well liked by his colleagues.” (p. 421)
What is the probability that Dick is a lawyer?
You probably guessed 50-50. That is wrong. You ignored the Base Rate of Lawyers. 30 out of these 100 men are lawyers. That means the Base Rate of Lawyers is 30%. That means the probability is 30% that Dick is a lawyer. The description doesn’t give you more evidence to make you think Dick has a 50% chance of being a lawyer.
How Can You Avoid the Representative Bias?
- Get the Outside View (also known as the Base Rate). To get the Outside View you must ignore detailed information about the case. Find the average rate of a case.
- Get the Inside View. To get the Inside View you must gather more information and details about a case.
- Estimate the quality and Validity of the Inside View.
- Now start with Outside View, and adjust it with the Inside View. Your prediction will be a number between the Outside View and the Inside View. If the quality and validity of the Inside View is weak, your prediction should be closer to the Outside View.
“There is one thing you can do when you have doubts about the quality of the evidence: let your judgments of probability stay close to the base rate. Don’t expect this exercise of discipline to be easy—it requires a significant effort of self-monitoring and self-control.” (p. 153)
How Can You Use Bayes’ Theorem to Calculate the Probability of Tom’s Major?
- 1. What is the Outside View (Base Rate) of each major? (“Some people ignore base rates because they believe them to be irrelevant in the presence of individual information. Others make the same mistake because they are not focused on the task”.) (p. 153)
- 2. Let’s say the Outside View of a Computer Science major is around 3%.
- 3. Now let’s check the Inside View. Let’s look at the description of Tom. Tom’s description sounds like he is more likely to be a computer science major.
- 4. Let’s say the description shows he’s 4X (400%) more likely than an average student to be a computer science major. So now you have the Inside View of 400% and Outside View of 3%.
- 5. By using Baye’s Theorem, it shows the probability of Tom being a computer science major is 11%.
Another Example:
“You see a person reading The New York Times on the New York subway. Which of the following is a better bet about the reading stranger?” (p. 151)
She has a PhD.
She does not have a college degree.
You probably said Phd. That’s the wrong answer.
How Can You Get a More Accurate Prediction?
- Get Outside View or Base Rate. Ignore the detailed information.
What is the Base Rate of a random person having a PhD? (Let’s guess1%)
What is the Base Rate of a random person having no college degree? (Let’s guess 33%)
- Now gather evidence or information for the Inside View. The Inside View is she reads the New York Times.
- Let’s say say you think 50% of all people with a PhD reads the NY times. So the Inside View is 50%.
So you start with the the Outside View of 1% and you anchor it with the Inside View of 50%. You final prediction that the person having a PhD should be 0.5%. - Let’s say you think only 10% of people with no college degree reads the NY Times. So the Inside View is 10%.
So you start with the Outside View of 33% and you anchor it with the Inside View of 10%. Your final prediction of someone that doesn’t have a college degree should be 3.3%.
“Representativeness would tell you to bet on the PhD, but this is not necessarily wise. You should seriously consider the second alternative, because many more nongraduates than PhDs ride in New York subways.” (p. 151-152)
Another Example:
- “if you must guess whether a woman who is described as “a shy poetry lover” studies Chinese literature or business administration, you should opt for the latter option. Even if every female student of Chinese literature is shy and loves poetry, it is almost certain that there are more bashful poetry lovers in the much larger population of business students.” (p. 152)
Example of Base Rate Neglect (From P. 166)
“Consider the following scenario and note your intuitive answer to the question.
A cab was involved in a hit-and-run accident at night. Two cab companies, the Green and the Blue, operate in the city. You are given the following data: 85% of the cabs in the city are Green and 15% are Blue.
A witness identified the cab as Blue. The court tested the reliability of the witness under the circumstances that existed on the night of the accident and concluded that the witness correctly identified each one of the two colors 80% of the time and failed 20% of the time.
What is the probability that the cab involved in the accident was Blue rather than Green?”
“The two sources of information can be combined by Bayes’s rule. The correct answer is 41%. However, you can probably guess what people do when faced with this problem: they ignore the base rate and go with the witness. The most common answer is 80%.” (p. 166)