Reality test your assumptions.

From Chip Heath’s “Decisive”

“Confirmation bias is probably the single biggest problem in business, because even the most sophisticated people get it wrong.”

• “Our normal habit in life is to develop a quick belief about a situation and then seek out information that bolsters our belief. And that problematic habit, called the “confirmation bias,” is the second villain of decision making.”
  

• “Researchers have found this result again and again. When people have the opportunity to collect information from the world, they are more likely to select information that supports their preexisting attitudes, beliefs, and actions. Example- Political partisans seek out media outlets that support their side but will rarely challenge their beliefs by seeking out the other side’s perspective.”
  
  
• “the confirmation bias is not easily disrupted. Even the smartest psychologists, who have studied the bias for years, admit that they can’t shake it. It can’t be wiped out; it can only be reined in. To see how we can hold our own against this tenacious foe, continue on to the next chapter, and get ready to Reality-Test Your Assumptions.” (p. 101)
  
  
• “Reality-Testing Our Assumptions is difficult. We’ll rarely do it instinctively. That’s the whole point of the confirmation bias—deep down, we never really want to hear the negative information. (When’s the last time you earnestly “considered the opposite” of one of your political views?) That’s why we are advocating so strongly in this book for the use of a process, something that becomes habitual. Otherwise it will be too easy to discard this advice in the heat of the moment.” (p. 126)
  
  
• “The meta-study found that the confirmation bias was stronger in emotion-laden domains such as religion or politics and also when people had a strong underlying motive to believe one way or the other (as in Upton Sinclair’s observation, “It is difficult to get a man to understand something when his salary depends on his not understanding it!”). The confirmation bias also increased when people had previously invested a lot of time or effort in a given issue.” (p. 108)
  
  
• “this is what’s slightly terrifying about the confirmation bias: When we want something to be true, we will spotlight the things that support it, and then, when we draw conclusions from those spotlighted scenes, we’ll congratulate ourselves on a reasoned decision. Oops.”
  
  
• “People go out and they’re collecting the data, and they don’t realize they’re cooking the books.”
  
  
• “Or pity the poor contestants who try out to sing on reality TV shows, despite having no discernible ability to carry a tune. When they get harsh feedback from the judges, they look shocked. Crushed. And you realize: This is the first time in their lives they’ve received honest feedback. Eager for reassurance, they’d locked their spotlights on the praise and support from friends and family. Given that affirmation, it’s not hard to see why they’d think they had a chance to become the next American Idol. It was a reasonable conclusion drawn from a wildly distorted pool of data.” (L. 192)

To counter confirmation bias, reality test your assumptions. Here are the tools they recommend.

Seek out disagreement.

• “Hayward and Hambrick also discovered an antidote to hubris: disagreement. They found that CEOs paid lower acquisition premiums when they had people around them who were more likely to challenge their thinking, such as an independent chairman of the board or outside board members who were unconnected to the CEO or the company.”
  
  
• “To make good decisions, CEOs need the courage to seek out disagreement. Alfred Sloan, the longtime CEO and chairman of General Motors, once interrupted a committee meeting with a question: “Gentlemen, I take it we are all in complete agreement on the decision here?” All the committee members nodded. “Then,” Sloan said, “I propose we postpone further discussion of this matter until our next meeting to give ourselves time to develop disagreement and perhaps gain some understanding of what this decision is about.” (p. 107)
  
  
• “In most legal systems, disagreement is baked into the process. Judges and juries will never find themselves in a CEO-style information bubble, since they are forced to consider two opposing points of view.”
  
  
• “In our individual decisions, how many of us have ever consciously sought out people we knew would disagree with us? Certainly not every decision needs a devil’s advocate—“I strenuously object to your purchasing those slacks!”—but for high-stakes decisions, we owe ourselves a dose of skepticism.” (p. 109)
  
  
• Create a Devils advocate.
  
  “There are many ways to honor that spirit of values-based opposition. In some organizations, the executive in charge might assign a few people on the executive team to prepare a case against a high-stakes proposal. That’s a wise idea. It puts the team members in the role of “protecting the organization,” and it licenses their skepticism.”
  
  “The most important lesson to learn about devil’s advocacy isn’t the need for a formal contrarian position; it’s the need to interpret criticism as a noble function.”
  
  “Another alternative is to seek out existing dissent rather than creating it artificially. If you haven’t encountered any opposition to a decision you’re considering, chances are you haven’t looked hard enough. Could you create a safe forum where critics can air their concerns?” (p. 110)
  
  
• “How can we plan for disagreement inside organizations? Some have created devil’s advocate–style traditions. The Pentagon used a “murder board,” staffed with experienced officers, to try to kill ill-conceived missions.”
  
  
• “That’s the whole point of the confirmation bias—deep down, we never really want to hear the negative information. (When’s the last time you earnestly “considered the opposite” of one of your political views?) That’s why we are advocating so strongly in this book for the use of a process, something that becomes habitual. Otherwise it will be too easy to discard this advice in the heat of the moment.”
  
  
• “What we’ve seen so far is a very simple rule for analyzing your options: Take the outside view. You should distrust the inside view—those glossy pictures in your head—and instead get out of your head and consult the base rates. Sometimes those numbers are readily available, as on TripAdvisor or Yelp. Sometimes you might have to cobble them together yourself. If neither of those options is possible, try consulting an expert for their estimates of the base rates.”

Ask yourself “What would have to be true for the option to be the right answer?”

• Roger Martin says the “What would have to be true?” question has become the most important ingredient of his strategy work, and it’s not hard to see why. The search for disconfirming information might seem, on the surface, like a thoroughly negative process: We try to poke holes in our own arguments or the arguments of others. But Martin’s question adds something constructive: What if our least favorite option were actually the best one? What data might convince us of that?” (p. 114)

Example of how this works when two parties can’t agree.

“As the discussion progressed, the two sides settled into their predictable roles: The executives were leaning toward closing the mine, and the mining managers opposed it. People were talking past one another. Roger Martin describes the initial discussion as “all over the map.” “I remember we’d been there probably a couple of hours,” said the treasurer, Ross. “And there was this sense of frustration. There’s a lot here to talk about. How do we work through it?” “I could tell it was going nowhere,” said Martin.

At the point of impasse, he said, “an idea popped into my head.” He issued the group a challenge: Let’s stop arguing about who is right, he said. Instead, let’s take each option, one at a time, and ask ourselves: What would have to be true for this option to be the right answer?

Surely it’s possible, he said, to imagine a set of evidence that would persuade us to change our minds. Let’s talk about what that evidence would look like. Ross said that after Roger Martin posed his challenge, “the lights went on for everyone. Participants switched from arguing to analyzing, discussing the logical underpinnings of each option.

The executives, asked to specify the conditions under which it would make sense to keep the mine open, started talking about production targets that would make it viable. The mine managers, asked to contemplate a scenario where closing the mine might be the best option, agreed that if copper prices didn’t recover, it would be hard to recommend continued operations.

The tenor of the discussion changed. There was still tension in the room, but it was productive tension. Martin’s reframing of the meeting had changed adversaries into collaborators. “It was magic,” said Martin. “By the end of the day, we had the group’s agreement on what had to be true for each of the five options for it to be the very best choice.” (p. 113)

Try to get disconfirming information.

• “Sometimes we think we’re gathering information when we’re actually fishing for support. Take the tradition of calling people’s references when you want to hire them. It’s an exercise in self-justification: We believe someone is worth hiring, and as a final “check” on ourselves, we decide to gather more information about them from past colleagues. So far, so good. Then we allow the candidate to tell us whom we should call, and we dutifully interview those people, who say glowing things about the candidate, and then, absurdly, we feel more confident in our decision to hire the person. (Imagine if we bought a time-share because the salesman had three awesome references.)”
  
  
• “This practice of asking probing questions is useful when you are trying to pry information from people who have an incentive to spin you: salesmen, recruiters, employees with agendas, and so on.”
  
  
• “Asking tough, disconfirming questions like these can dramatically improve the quality of information we collect“
  
  
• “For example, Ray Rothrock, a venture capitalist with Venrock, says that one of the best diagnostic questions he’s discovered in assessing entrepreneurs is “How many secretaries has this entrepreneur had in the past few years?” If the answer is five, chances are you’ve got someone with some issues.”
  
  
• “When you are at a recruiting dinner with a couple of lawyers from the firm, don’t just ask them, “So, do you folks have any kind of life outside of work?” They will chuckle, say “sure,” and ask if you want more wine. Instead, ask them how many times last week they had dinner with their families. And then ask them what time dinner was served. And then ask them whether they worked after dinner.
  
  Ask them what their favorite television show is or what is the last good movie they saw. If they respond, respectively, Welcome Back, Kotter and Saturday Night Fever, you will know something’s wrong.… When a lawyer tells you that he gets a lot of interesting assignments, ask for examples. You may be surprised at what passes for “interesting” at the firm.”
  
  And when a lawyer tells you that associates are happy at the firm, ask for specifics. How many associates were hired five years ago? How many of those associates remain at the firm? Who were the last three associates to leave the firm? What are they doing now? How can you contact them?” (p. 116)
  
  
• Ask,”What problems does it have?”
  
  “The researchers wondered what it would take for the sellers to disclose the freezing problem. The buyers in the negotiation, who were cronies of the researchers, tried three different strategies. When the buyers asked about the iPod, “What can you tell me about it?,” only 8% of the sellers disclosed the problem. The question “It doesn’t have any problems, does it?” boosted the disclosure to 61%. The best question to ask, in hopes of discovering the truth, was this one: “What problems does it have?” That prompted 89% of the sellers to come clean.”
  
  
• “In some organizations, hiring managers have become smarter about reference calls. Some ask the references for additional people to contact who weren’t on the original list. Those secondary interviews will tend to yield more neutral information. Other people have reconsidered the kinds of questions they ask in reference calls. Rather than ask for an evaluation of the candidate (“Would you say Steve’s performance was closer to ‘stunning’ or ‘breathtaking’? Be honest.”), many firms now seek specific factual information.”
  
  
• “How do you know whether to ask probing questions or open-ended ones? A good rule of thumb is to ask yourself, “What’s the most likely way I could fail to get the right information in this situation?” Generally, it will be obvious what the answer is: If you’re buying a used car, you’re most likely to fail by not discovering a flaw of the vehicle, or if you’re a vice president seeking feedback from factory workers, you’re most likely to fail by not uncovering what they really think. You can tailor your questions accordingly—more aggressive in the used-car negotiations and more open-ended with the factory worker.” (p. 121)

Extreme disconfirmation: Can we force ourselves to consider the opposite of our instincts?

• Example: There is a couple having probles. Therapists suggest they keep a marriage diary. The couple will write down good things their partner have done. This will help each spouse notice all the good things their partner has done, instead of thinking about the negative things their partner has done.

When gathering information use the “outside view.” Click here for more details.

• “The outside view is more accurate—it’s a summary of real-world experiences, rather than a single person’s impressions—yet we’ll be drawn to the inside view.”
  
  
• “What we’ve seen so far is a very simple rule for analyzing your options: Take the outside view. You should distrust the inside view—those glossy pictures in your head—and instead get out of your head and consult the base rates. Sometimes those numbers are readily available, as on TripAdvisor or Yelp. Sometimes you might have to cobble them together yourself. If neither of those options is possible, try consulting an expert for their estimates of
  the base rates.”
  
  
• “This brings us to a critical point about experts. Perhaps the simplest and most intuitive advice we can offer in this chapter is that when you’re trying to gather good information and reality-test your ideas, go talk to an expert. If you’re considering filing an intellectual-property lawsuit against a competitor, talk to a top IP lawyer. An expert doesn’t have to be a heavily credentialed authority, though. The bar is actually far lower than that: An expert is simply someone who has more experience than you. If your son wants to be a carpenter, go talk to a carpenter. Any carpenter. If you’re thinking about relocating your business to South Carolina, call up someone, anyone, who has relocated their business to South Carolina.” (p. 136)

Get a close up look.

• A “close-up” can add texture that’s missing from the outside view.
  
  
• ” Brian Zikmund-Fisher studied the base-rate outcomes of patients with MDS, but he also sought a close-up (discovering the need for exercise and a “third adult”)
  
  
• FDR had his staff compile a statistical “mail brief,” and he also read a sample of the letters. More close-ups: Xerox’s customer officer of the day. “Going to the genba.” Using competitors’ paper towels. (p. 153)

Try Ooching.

• “To ooch is to construct small experiments to test one’s hypothesis.” (p. 116)
  
• Ooching = “running small experiments to test our theories. Rather than jumping in headfirst, we dip a toe in.”
  
• “That strategy—finding a way to ooch before we leap—is another way we can reality-test our assumptions. When we ooch, we bring real-world experience into our decision.”
  
• “To ooch is to ask, Why predict something we can test? Why guess when we can know?”

Examples:

• Carsdirect.com quickly put a website up to see if they could sell cars. “In the brainstorming session there was a lot of resistance because some thought it was unlikely that people would buy a big-ticket item like that through the Web. At that time no one was selling cars through the Web. So rather than continue debating it, we put up a Web site with a couple of pages that looked like it would allow you to order a car. But actually the message went to a clerk, who looked up the price in the Kelley Blue Book and sent it back to the user. The next morning Bill discovered we had sold three cars. We had to quickly shut down the site because we were offering a heavy discount on the cars. Rather than continuing to debate, the team ooched and resolved the uncertainty. The ooch led to the founding of CarsDirect.com,”
  
  
• “In trying to meet the biologists’ needs, Hanks’s team didn’t bother building an elegant product. Elegance is expensive and time-consuming. Instead, they cobbled together a prototype using what they had on hand. Hanks compared the result to a “brick in a bucket.”
  
  
• “Rather than jump headfirst into the wireless market, Hanks and his colleagues decided to dip a toe in. Rather than choose “all” or “nothing,” they chose “a little something.” That strategy—finding a way to ooch before we leap—is another way we can reality-test our assumptions. When we ooch, we bring real-world experience into our decision.”
  
  
• “the notion of exploring options with small experiments has popped up in many different places. Designers talk about “prototyping”; rather than spending six months planning the perfect product, they’ll just hack together a quick mock-up and get it in the hands of potential customers.”
  
  
• Steve has decided to become a Pharmacist. How did he decide to become a pharmacist?
  
  “He’s always enjoyed chemistry, after all, and he likes the idea of working in health care. He feels like the lifestyle of a pharmacist, with its semireasonable hours and good pay, would suit him well.
  
  But this is pretty thin evidence for such an important decision! Steve is contemplating a minimum time commitment of two years for graduate school, not to mention tens of thousands of dollars in tuition and forgone income. He’s placing a huge bet on paltry information. This is a situation that cries out for an ooch, and an obvious one would be to work in a pharmacy for a few weeks. He’d be smart to work for free, if need be, to get the job. (Certainly if he can afford several years of school without an income, he can afford to take a monthlong unpaid internship.)” (p. 155)
  
  “Surely this concept—testing a profession before entering it—sounds obvious. Yet every year hordes of students enroll in graduate schools without ever having run an experiment like that: law students who’ve never spent a day in a law office and med students who’ve never spent time in a hospital or clinic. Imagine going to school for three or four years so you can start a career that never suited you! This is a truly terrible decision process, in the same league as an impromptu drunken marriage in Vegas.” (p. 156)
  
  
• “HopeLab has begun to give potential employees a three-week consulting contract. Cole said, “It’s unbelievably effective. No more fear. How are we going to make our hiring decisions? We make our decisions based on the empirical performance of the employee in our community, on the kinds of jobs that we do. The job market totally prevents you from getting this kind of useful information. So collect your own personal performance data in your own personal context. In some ways it really doesn’t matter how well they did in their last job.”
  
  “Next time you’ve got a job opening to fill, consider Steve Cole’s advice. What’s the best way you could give your potential hires a trial run?“