From Kahneman’s “Noise”
Why are Traditional Interviews Bad?
- “if your goal is to determine which candidates will succeed in a job and which will fail, standard interviews (also called unstructured interviews to distinguish them from structured interviews, to which we will turn shortly) are not very informative. To put it more starkly, they are often useless.” (p. 280)
- “There is overwhelming evidence of the superiority of structured judgment processes (including structured interviews) in hiring.” “Nevertheless, most executives remain convinced of the irreplaceable value of informal, interview-based methods. Remarkably, so do many candidates who believe that only a face-to-face interview will enable them to show a prospective employer their true mettle. Researchers have called this “the persistence of an illusion.” One thing is clear: recruiters and candidates severely underestimate the noise in hiring judgments.” (p. 289)
- “In chapter 11, we mentioned a correlation between typical interview ratings and job performance ratings of .28. Other studies report correlations that range between .20 and .33. As we have seen, this is a very good correlation by social science standards—but not a very good one on which to base your decisions.” (p. 280)
- “Why do first impressions end up driving the outcome of a much longer interview? One reason is that in a traditional interview, interviewers are at liberty to steer the interview in the direction they see fit. They are likely to ask questions that confirm an initial impression.” (p. 283)
.
How to Conduct a Structured Interview.
“a structured complex judgment is defined by three principles: decomposition, independence, and delayed holistic judgment.” (p. 286)
- “The first principle, decomposition, breaks down the decision into components, or mediating assessments. This step serves the same purpose as the identification of the subjudgments in a guideline: it focuses the judges on the important cues. Decomposition acts as a road map to specify what data is needed. And it filters out irrelevant information.” (p. 286)
- “In Google’s case, there are four mediating assessments in the decomposition: general cognitive ability, leadership, cultural fit (called “googleyness”), and role-related knowledge. (Some of these assessments are then broken down into smaller components.)” (p. 286)
- “The second principle of structured judgment, independence, requires that information on each assessment be collected independently. Just listing the components of the job description is not enough: most recruiters conducting traditional interviews also know the four or five things they look for in a candidate. The problem is that, in the conduct of the interview, they do not evaluate these elements separately. Each assessment influences the others, which makes each assessment very noisy.” (p. 286)
- “To overcome this problem, Google orchestrated ways to make assessments in a fact-based manner and independently of one another. Perhaps its most visible move was to introduce structured behavioral interviews. The interviewers’ task in such interviews is not to decide whether they like a candidate overall; it is to collect data about each assessment in the evaluation structure and to assign a score to the candidate on each assessment. To do so, interviewers are required to ask predefined questions about the candidate’s behaviors in past situations. They must also record the answers and score them against a predetermined rating scale, using a unified rubric. The rubric gives examples of what average, good, or great answers look like for each question. This shared scale (an example of the behaviorally anchored rating scales we introduced in the preceding chapter) helps reduce noise in judgments.” (p. 287)
- “The third principle of structured judgment, delayed holistic judgment, can be summarized in a simple prescription: do not exclude intuition, but delay it.” (p. 288)
- “Google’s final hiring decisions are anchored on the average score assigned by the four interviewers. They are also informed by the underlying evidence. In other words, Google allows judgment and intuition in its decision-making process only after all the evidence has been collected and analyzed.” (p. 288)
Other Good Quotes
“One of the conclusions of Bock’s review was to reduce that number to four, as he found that additional interviews added almost no predictive validity to what was achieved by the first four.” (p. 285)
“As professional recruiters know, however, defining the key assessments gets difficult for unusual or senior positions, and this step of definition is frequently overlooked.” (p. 286)
“Google stringently enforces a rule that not all companies observe: the company makes sure that the interviewers rate the candidate separately, before they communicate with one another. Once more: aggregation works—but only if the judgments are independent.” (p. 285)
“One prominent headhunter points out that defining the required competencies in a sufficiently specific manner is a challenging, often overlooked task. He highlights the importance for decision makers of “investing in the problem definition”: spending the necessary time up front, before you meet any candidates, to agree on a clear and detailed job description.” (p. 2860
“Google uses other data as inputs on some of the dimensions it cares about. To test job-related knowledge, it relies in part on work sample tests, such as asking a candidate for a programming job to write some code. Research has shown that work sample tests are among the best predictors of on-the-job performance. Google also uses “backdoor references,” supplied not by someone the candidate has nominated but by Google employees with whom the candidate has crossed paths.” (p. 287)