From Kahneman’s “Noise”
How to Lessen “Noise”
- Select better judges. Click here to see how.
- “A radical application of this principle is the replacement of judgment with rules or algorithms. Algorithmic evaluation is guaranteed to eliminate noise—indeed, it is the only approach that can eliminate noise completely.” (p. 345)
- Aggregate multiple independent estimates.
“Let’s take the average of four independent judgments—this is guaranteed to reduce noise by half.” (p. 253)
“The easiest way to aggregate several forecasts it to average them. Averaging is mathematically guaranteed to reduce noise” (p. 242)
“Straight averaging is not the only way to aggregate forecasts. A select-crowd strategy, which selects the best judges according to the accuracy of their recent judgments and averages the judgments of a small number of judges (e.g., five), can be as effective as straight averaging.” (p. 243) - Use the mini-Dephi method. “Also called estimate-talk-estimate, it requires participants first to produce separate (and silent) estimates, then to explain and justify them, and finally to make a new estimate in response to the estimates and explanations of others.” (p. 243)
- Train and think like a Superforecaster. Think in probabilities and use the Outside View. Superforecasters take a training tutorial about Biases that includes Base-rate neglect, Overconfidence, and Confirmation Bias.
- “Teaming (a form of aggregation): Some forecasters were asked to work in teams in which they could see and debate one another’s predictions. Teaming could increase accuracy by encouraging forecasters to deal with opposing arguments and to be actively open-minded.” (p. 249)
- Team up with the best forecasters. “Teaming had a comparably large effect on noise reduction, but it also significantly improved the ability of the teams to extract information. This result is consistent with the logic of aggregation: several brains that work together are better at finding information than one is.” (p. 251) Be actively openminded. Use probabilities.
- “Suppose that a particular forecaster named Margaret says that 500 different events are 60% likely. If 300 of them actually happen, then we can conclude that Margaret’s confidence is well calibrated. Good calibration is one requirement for good forecasting.” (p. 245)
- “Think statistically, and take the outside view of the case. We say that a judge takes the outside view of a case when she considers it as a member of a reference class of similar cases rather than as a unique problem.” (p. 346)
- Use the Outside View. “Superforecasters also excel at taking the outside view, and they care a lot about base rates. As explained for the Gambardi problem in chapter 13, before you focus on the specifics of Gambardi’s profile, it helps to know the probability that the average CEO will be fired or quit in the next two years. Superforecasters systematically look for base rates.” (p. 248)
- Update your views and update your forecasts. “This approach mirrors what is expected of forecasters in business and government, who should also be updating their forecasts frequently on the basis of new information, despite the risk of being criticized for changing their minds.” (p. 246)
- “People can reduce excessive coherence by breaking down the judgment problem into a series of smaller tasks. This technique is analogous to the practice of structured interviews, in which interviewers evaluate one trait at a time and score it before moving to the next one.” (p. 346)
- “The principle of structuring inspires diagnostic guidelines, such as the Apgar score. It is also at the heart of the approach we have called the mediating assessments protocol. This protocol breaks down a complex judgment into multiple fact-based assessments and aims to ensure that each one is evaluated independently of the others. Whenever possible, independence is protected by assigning assessments to different teams and minimizing communication among them.” (p. 346)
- “Resist premature intuitions. We have described the internal signal of judgment completion that gives decision makers confidence in their judgment. The unwillingness of decision makers to give up this rewarding signal is a key reason for the resistance to the use of guidelines and algorithms and other rules that tie their hands. Decision makers clearly need to be comfortable with their eventual choice and to attain the rewarding sense of intuitive confidence. But they should not grant themselves this reward prematurely. An intuitive choice that is informed by a balanced and careful consideration of the evidence is far superior to a snap judgment.” (p. 347)
- “Obtain independent judgments from multiple judges, then consider aggregating those judgments. The requirement of independence is routinely violated in the procedures of organizations, notably in meetings in which participants’ opinions are shaped by those of others. Because of cascade effects and group polarization, group discussions often increase noise.” (p. 347)
- “Favor relative judgments and relative scales. Relative judgments are less noisy than absolute ones, because our ability to categorize objects on a scale is limited, while our ability to make pairwise comparisons is much better. Judgment scales that call for comparisons will be less noisy than scales that require absolute judgments.” (p. 348)