From Daniel Kahneman’s “Noise”
My Favorite Quotes From “Noise”
“wherever there is judgment, there is noise—and more of it than you think.” (p. 16)
“Experiments show large disparities among judges in the sentences they recommend for identical cases. This variability cannot be fair. A defendant’s sentence should not depend on which judge the case happens to be assigned to.” “Criminal sentences should not depend on the judge’s mood during the hearing, or on the outside temperature.” (p. 25)
“Most executives of the insurance company guessed 10% or less. When we asked 828 CEOs and senior executives from a variety of industries how much variation they expected to find in similar expert judgments, 10% was also the median answer and the most frequent one (the second most popular was 15%). (p. 29).
Our noise audit found much greater differences. By our measure, the median difference in underwriting was 55%,” (p. 29)
“Variability in judgments is also expected and welcome in a competitive situation in which the best judgments will be rewarded. When several companies (or several teams in the same organization) compete to generate innovative solutions to the same customer problem, we don’t want them to focus on the same approach. The same is true when multiple teams of researchers attack a scientific problem, such as the development of a vaccine: we very much want them to look at it from different angles.” (p. 30)
“Other people view the world much the way I do.” These beliefs, which have been called naive realism, are essential to the sense of a reality we share with other people. We rarely question these beliefs. We hold a single interpretation of the world around us at any one time, and we normally invest little effort in generating plausible alternatives to it.” (p. 33)
“The school’s reply: “We used to do that, but it resulted in so many disagreements that we switched to the current system.” (p. 34)
“Whether you make a decision only once or a hundred times, your goal should be to make it in a way that reduces both bias and noise. And practices that reduce error should be just as effective in your one-of-a-kind decisions as in your repeated ones.” (p. 40)
“One approach to the evaluation of the process of judgment is to observe how that process performs when it is applied to a large number of cases. For instance, consider a political forecaster who has assigned probabilities of winning to a large number of candidates in local elections. He described one hundred of these candidates as being 70% likely to win. If seventy of them are eventually elected, we have a good indication of the forecaster’s skill in using the probability scale.” (p. 50)
“Focusing on the process of judgment, rather than its outcome, makes it possible to evaluate the quality of judgments that are not verifiable, such as judgments about fictitious problems or long-term forecasts.” (p. 50)
“Scholars of decision-making offer clear advice to resolve this tension: focus on the process, not on the outcome of a single case. We recognize, however, that this is not standard practice in real life. Professionals are usually evaluated on how closely their judgments match verifiable outcomes, and if you ask them what they aim for in their judgments, a close match is what they will answer.” (p. 51)
“Simply put, just like a basketball player who never throws the ball twice in exactly the same way, we do not always produce identical judgments when faced with the same facts on two occasions.” (p. 79)
“The reason is basic statistics: averaging several independent judgments (or measurements) yields a new judgment, which is less noisy, albeit not less biased, than the individual judgments.” (p. 82)
“First, assume that your first estimate is off the mark. Second, think about a few reasons why that could be. Which assumptions and considerations could have been wrong? Third, what do these new considerations imply? Was the first estimate rather too high or too low? Fourth, based on this new perspective, make a second, alternative estimate.” (p. 83)
“Mood, fatigue, weather, sequence effects: many factors may trigger unwanted variations in the judgment of the same case by the same person.” (p. 89)
“Information is not, of course, the only reason that group members are influenced by one another. Social pressures also matter. At a company or in government, people might silence themselves so as not to appear uncongenial, truculent, obtuse, or stupid. They want to be team players.” (p. 99)
“The potential dependence of outcomes on the judgments of a few individuals—those who speak first or who have the largest influence—should be especially worrisome now that we have explored how noisy individual judgments can be.” (p. 102)
“the fact that most judgments are made in a state of what we call objective ignorance, because many things on which the future depends can simply not be known. Strikingly, we manage, most of the time, to remain oblivious to this limitation and make predictions with confidence (or, indeed, overconfidence).” (p. 106)
“As we noted, clinical predictions achieved a .15 correlation (PC = 55%) with job performance, but mechanical prediction achieved a correlation of .32 (PC = 60%). Think about the confidence that you experienced in the relative merits of the cases of Monica and Nathalie. Meehl’s results strongly suggest that any satisfaction you felt with the quality of your judgment was an illusion: the illusion of validity.” (p. 111)
“You can often be quite confident in your assessment of which of two candidates looks better, but guessing which of them will actually be better is an altogether different kettle of fish.” (p. 111)
“Meehl’s pattern contradicts the subjective experience of judgment, and most of us will trust our experience over a scholar’s claim.” (p. 111)
“But you can surely imagine your own dismay if someone told you that a crude model of your judgments—almost a caricature—was actually more accurate than you were. For most of us, the activity of judgment is complex, rich, and interesting precisely because it does not fit simple rules.” (p. 114)
“the gains from subtle rules in human judgment—when they exist—are generally not sufficient to compensate for the detrimental effects of noise. You may believe that you are subtler, more insightful, and more nuanced than the linear caricature of your thinking. But in fact, you are mostly noisier.” (p. 115)
“People believe they capture complexity and add subtlety when they make judgments. But the complexity and the subtlety are mostly wasted—usually they do not add to the accuracy of simple models.” (p. 117)
“In predictive judgments, human experts are easily outperformed by simple formulas—models of reality, models of a judge, or even randomly generated models.” (p. 117)
“Many experts ignore the clinical-versus-mechanical debate, preferring to trust their judgment. They have faith in their intuitions and doubt that machines could do better. They regard the idea of algorithmic decision making as dehumanizing and as an abdication of their responsibility.” (p. 128)
“More often, people are willing to give an algorithm a chance but stop trusting it as soon as they see that it makes mistakes.” (p. 128)
“In general, however, you can safely expect that people who engage in predictive tasks will underestimate their objective ignorance. Overconfidence is one of the best-documented cognitive biases. In particular, judgments of one’s ability to make precise predictions, even from limited information, are notoriously overconfident. What we said of noise in predictive judgments can also be said of objective ignorance: wherever there is prediction, there is ignorance, and more of it than you think.” (p. 133)
“When you trust your gut because of an internal signal, not because of anything you really know, you are in denial of your objective ignorance.” (p. 138)
“The distinction between these two views is a recurring theme of this book. Relying on causal thinking about a single case is a source of predictable errors. Taking the statistical view, which we will also call the outside view, is a way to avoid these errors.” (p. 148)
“Causal thinking helps us make sense of a world that is far less predictable than we think. It also explains why we view the world as far more predictable than it really is. In the valley of the normal, there are no surprises and no inconsistencies. The future seems as predictable as the past. And noise is neither heard nor seen.” (p. 149)
“For instance, when people forecast how long it will take them to complete a project, the mean of their estimates is usually much lower than the time they will actually need. This familiar psychological bias is known as the planning fallacy.” (p. 152)
“Taking the outside view can make a large difference and prevent significant errors. A few minutes of research would reveal that estimates of CEO turnover in US companies hover around 15% annually. This statistic suggests that the average incoming CEO has a roughly 72% probability of still being around after two years. Of course, this number is only a starting point, and the specifics of Gambardi’s case will affect your final estimate. But if you focused solely on what you were told about Gambardi, you neglected a key piece of information.” (p. 156)
“Regardless of the question, substituting one question for another will lead to an answer that does not give different aspects of the evidence their appropriate weights, and incorrect weighting of the evidence inevitably results in error. For example, a full answer to a question about life satisfaction clearly requires consulting more than your current mood, but evidence suggests that mood is in fact overly weighted.” (p. 157)
“Prejudgments are evident wherever we look. Like Lucas’s reaction, they often have an emotional component. The psychologist Paul Slovic terms this the affect heuristic: people determine what they think by consulting their feelings.” (p. 159)
“This experiment illustrates excessive coherence: we form coherent impressions quickly and are slow to change them. In this example, we immediately developed a positive attitude toward the candidate, in light of little evidence.” (p. 161)
“In general, we jump to conclusions, then stick to them. We think we base our opinions on evidence, but the evidence we consider and our interpretation of it are likely to be distorted, at least to some extent, to fit our initial snap judgment. As a result, we maintain the coherence of the overall story that has emerged in our mind.” (p. 161)
“When the question “Is there climate change?” is replaced with “Do I trust the people who say it is real?,” (p. 162)
“The outside view can be neglected only in very easy problems, when the information available supports a prediction that can be made with complete confidence. When serious judgment is necessary, the outside view must be part of the solution.” (p. 171)
“Most people are surprised to hear that the accuracy of their predictive judgments is not only low but also inferior to that of formulas. Even simple linear models built on limited data, or simple rules that can be sketched on the back of an envelope, consistently outperform human judges.” (p. 342)
“To be clear, personal values, individuality, and creativity are needed, even essential, in many phases of thinking and decision making, including the choice of goals, the formulation of novel ways to approach a problem, and the generation of options. But when it comes to making a judgment about these options, expressions of individuality are a source of noise.” (p. 345)