“Superthinking,” by Gabriel Weinberg
Lies, Damned Lies, and Statistics
“It is human nature to use past experience and observation to guide decision making, and evolutionarily this makes sense. If you watched someone get sick after they ate a certain food or get hurt by behaving a certain way around an animal, it follows that you should not copy that behavior. Unfortunately, this shortcut doesn’t always result in good thinking.”
“My grandfather lived to his eighties and smoked a pack a day for his whole life, so I don’t believe that smoking causes cancer.”
“I have heard several news reports about children being harmed. It is so much more dangerous to be a child these days.”
“These are all examples of drawing incorrect conclusions using anecdotal evidence, informally collected evidence from personal anecdotes. You run into trouble when you make generalizations based on anecdotal evidence or weigh it more heavily than scientific evidence. Unfortunately, as Michael Shermer, founder of the Skeptics Society, points out in his 2011 book The Believing Brain, “Anecdotal thinking comes naturally, science requires training.”
“Just because two events happened in succession, or are correlated, doesn’t mean that the first actually caused the second. Statisticians use the phrase correlation does not imply causation to describe this fallacy.”
“What is often overlooked when this fallacy arises is a confounding factor, a third, possibly non-obvious factor that influences both the assumed cause and the observed effect, confounding the ability to draw a correct conclusion. In the case of the flu vaccine, the cold and flu season is that confounding factor. People get the flu vaccine during the time of year when they are more likely to get sick, whether they have received the vaccine or not. Most likely the symptoms people are experiencing are from a common cold, which the flu vaccine does not protect against.”
“If you set out to collect or evaluate scientific evidence based on an experiment, the first step is to define or understand its hypothesis, the proposed explanation for the effect being studied (e.g., drinking Snapple can reduce the length of the common cold). Defining a hypothesis up front helps to avoid the Texas sharpshooter fallacy. This model is named after a joke about a person who comes upon a barn with targets drawn on the side and bullet holes in the middle of each target. He is amazed at the shooter’s accuracy, only to find that the targets were drawn around the bullet holes after the shots were fired.”
“One method to consider, often referred to as the gold standard in experimental design, is the randomized controlled experiment, where participants are randomly assigned to two groups, and then results from the experimental group (who receive a treatment) are compared with the results from the control group (who do not). This setup isn’t limited to medicine; it can be used in fields such as advertising and product development. (We will walk through a detailed example in a later section.) A popular version of this experimental design is A/B testing, where user behavior is compared between version A (the experimental group) and version B (the control group) of a site or product, which may differ in page flow, wording, imagery, colors, etc. Such experiments must be carefully designed to isolate the one factor you are studying. The simplest way to do this is to change just one thing between the two groups.”
“To take the idea of blinding one step further, the people administering the experiment or analyzing the experiment can also remain unaware of which group the participants are in. This additional blinding helps reduce the impact of observer-expectancy bias (also called experimenter bias), where the cognitive biases of the researchers, or observers, may cause them to influence the outcome in the direction they expected.”
“Interestingly, just the act of receiving something that you expect to have a positive effect can actually create one, called the placebo effect.”
“However, despite the complications that arise when conducting well-run experiments, collecting real scientific evidence beats anecdotal evidence hands down because you can draw believable conclusions. Yes, you have to watch out for spurious correlations and subtle biases (more on that in the next section), but in the end you have results that can really advance your thinking.”
“Selection bias can also occur when a sample is selected that is not representative of the broader population of interest, as with online reviews. If the group studied isn’t representative, then the results may not be applicable overall.”
“is the school better because there are better teachers or because the students are better prepared due to their parents’ financial means and interest in education? Selection bias likely explains some significant portion of these schools’ better test scores and college admissions.”
“Another type of selection bias, common to surveys, is nonresponse bias, which occurs when a subset of people don’t participate in an experiment after they are selected for it, e.g., they fail to respond to the survey. If the reason for not responding is related to the topic of the survey, the results will end up biased.”
“Surveys like this also do not usually account for the opinions of former employees, which can create another bias in the results called survivorship bias. Unhappy employees may have chosen to leave the company, but you cannot capture their opinions when you survey only current employees.”
“When you critically evaluate a study (or conduct one yourself), you need to ask yourself: Who is missing from the sample population? What could be making this sample population nonrandom relative to the underlying population? For example, if you want to grow your company’s customer base, you shouldn’t just sample existing customers; that sample doesn’t account for the probably much larger population of potential customers. This much larger potential customer base may behave very differently from your existing customer base”
“response bias. While nonresponse bias is introduced when certain types of people do not respond, for those who do respond, various cognitive biases can cause them to deviate from accurate or truthful responses. For example, in the employee engagement survey, people may lie (by omission or otherwise) for fear of reprisal.”
“BE WARY OF THE “LAW” OF SMALL NUMBERS When you interpret data, you should watch out for a basic mistake that causes all sorts of trouble: overstating results from a sample that is too small.”
“The name is derived from a valid statistical concept called the law of large numbers, which states that the larger the sample, the closer your average result is expected to be to the true average.”
“gambler’s fallacy, named after roulette players who believe that a streak of reds or blacks from a roulette wheel is more likely to end than to continue with the next spin.”
“You might be familiar with the phrase sophomore slump, which describes scenarios such as when a band gets rave reviews for their first album and the second one isn’t as well received, or when a baseball player has a fantastic rookie season but the next year his batting average is not that impressive. In these situations, you may assume there must be some psychological explanation, such as caving under the pressure of success. But in most cases, the true cause is purely mathematical, explained through a model called regression to the mean.”
“The takeaway is that you should never assume that a result based on a small set of observations is typical. It may not be representative of either another small set of observations or a much larger set of observations. Like anecdotal evidence, a small sample tells you very little beyond that what happened was within the range of possible outcomes. While first impressions can be accurate, you should treat them with skepticism. More data will help you distinguish what is likely from what is an anomaly.”
“This range has a corresponding confidence level, which quantifies the level of confidence you have that the true value of parameter is in the range you estimated. For example, a confidence level of 95 percent tells you that if you ran the poll many times and calculated many confidence intervals (one for each poll), on average 95 percent of them would include the true approval rating (i.e., 25 percent).”
“When a probability calculation fails to account for the base rate (like the base rate of drunk drivers), the mistake that is made is called the base rate fallacy.”
“Bayes’ theorem, which tells us the relationship between these two conditional probabilities.”
“Bayesians contend that by choosing a strong prior, they can start closer to the truth, allowing them to converge on the final result faster, with fewer observations.”
“Some but not all systematic reviews include meta-analyses, which use statistical techniques to combine data from several studies into one analysis. The data-driven reporting site FiveThirtyEight is a good example; it conducts meta-analyses across polling data to better predict political outcomes.”