Potential Problems with Sampling

In 1936 the magazine Literary Digest polled about 10 million Americans on the presidential race, got back about 2.4 million ballots, and predicted that Republican Alf Landon would beat Franklin Roosevelt with about 57% of the vote. Roosevelt won about 61% of the popular vote and 46 of the 48 states. The poll was huge and still wrong by a landslide, while George Gallup, with a sample of about 50,000, called the winner correctly. Size could not save the Digest, because its error came from how the sample was gathered, and no amount of extra data fixes that.

Bias is a property of the method

A statistic (a number computed from a sample, like a sample proportion) estimates a parameter (the same number for the whole population). Two different things can make a statistic miss.

Sampling variability is the ordinary chance difference between one random sample and the next. It goes both ways, high on some samples and low on others, and it shrinks as the sample gets larger.

Bias is a systematic error built into the sampling method: repeated over and over, the method produces statistics that are consistently too large, or consistently too small, compared with the parameter. A biased method misses in the same direction every time, and a larger sample does nothing about it.

A concrete case: in a town, 30% of adults are under 35. Of them, 70% stream music daily; of the older 70%, only 30% do. The true proportion who stream daily is

A pollster draws random numbers from the town's landline listings, where people under 35 make up only 10% of the names. Samples from that list center on

eight points too low. Drawing 2,000 people from the list instead of 200 just pins the estimate more tightly around the wrong value, 0.34.

Bias describes the method, not one sample. A single SRS might, by bad luck, contain too many seniors; that is sampling variability, and the next SRS might contain too few. A method is biased when its errors lean one way across the samples it would produce.

Bias from who ends up in the sample

Nonrandom sampling methods let people, not chance, decide who is in the sample, and that opens the door to bias. Two are common.

A convenience sample takes whoever is easiest to reach: the students in the cafeteria at noon, shoppers at one mall entrance, the researcher's own classmates. The individuals who are easy to reach often differ from the rest of the population in ways related to the question.

A voluntary response sample is made up entirely of people who chose to respond to an open invitation: a call-in radio poll, an online star rating, a link posted on social media. The resulting voluntary response bias comes from who bothers. People with strong opinions, usually negative ones, respond far more often than people who feel mildly. A restaurant's online reviews pile up at one star and five stars because diners who had an ordinary meal rarely write anything.

Even with random selection, the sampling method itself can miss people.

Undercoverage occurs when some part of the population cannot be selected by the method, or is less likely to be selected than others. The landline list above undercovers young adults. A survey of a city's residents drawn from property tax records misses renters, since the records list owners. A survey given during one class period misses every student who is not in school at that time. Undercoverage creates bias when the missed group answers differently from everyone else; the Digest's 1936 mailing list came largely from its own subscribers, telephone directories and automobile registrations, all of which skewed toward wealthier voters during the Depression.

Nonresponse occurs when individuals who were chosen for the sample can't be reached or decline to answer. Nonresponse creates bias when the people who respond differ from those who don't in ways that matter for the question. A survey on how many hours people work each week will hear least from the people working the most hours, because they have the least time to answer. Only about 24% of the Digest's 10 million ballots came back, and a 1988 analysis by political scientist Peverill Squire concluded that nonresponse was the bigger of its two problems: voters who disliked Roosevelt were more eager to mail in a ballot.

The difference between nonresponse and voluntary response is who chose. In voluntary response, the respondents selected themselves. In nonresponse, the researcher selected the sample and some of those people did not answer.

Bias from how responses are measured

Response bias occurs when the answers or measurements recorded tend to differ from the true values in one direction. The sample can be perfectly chosen and still produce biased data. Common sources:

• Question wording. Leading or confusing questions push answers. "Given the rising number of pedestrian injuries, do you support lower speed limits downtown?" steers respondents toward yes before they have considered the cost. A double negative ("Do you oppose not renewing the library levy?") confuses people into answering the opposite of what they mean.
• Self-reported answers. People shade their answers toward what sounds good or what they wish were true: they report more exercise, more voting and fewer drinks than they had. The bias grows when the question is sensitive and the setting is not private, such as a face-to-face interview.
• The interviewer or the instrument. Respondents may tell an interviewer what they think the interviewer wants to hear. A bathroom scale that reads 0.4 kg heavy makes every measured weight 0.4 kg too high, so the mean of 50 measured weights is also 0.4 kg too high, and the mean of 5,000 would be too.

Worked examples

Where students lose points

On the exam

Multiple-choice questions describe a survey and ask which bias is most likely, or which change would reduce it. In free response, part of the first question (which focuses on formulating questions and collecting data) can ask for a source of bias and its likely effect; the explanation must mention the context of the study and, when asked, a direction. Random selection removes the bias that comes from a person choosing the sample, but it does nothing about undercoverage in the list, nonresponse, or response bias, so each needs its own check.