Students who drink coffee before a test might score higher than students who don't. That alone says nothing about coffee: the coffee drinkers might also be the ones who got up early, ate breakfast, or care more about grades. An experiment removes those alternatives by deciding, with chance, who gets the coffee. Topic 1.10 set up the vocabulary (experimental units, explanatory variables or factors, treatments, and the response).
The four principles of a well-designed experiment
An extraneous variable is any variable that is known or believed to affect the response but is not one of the explanatory variables being studied. In a study of whether a new fertilizer increases tomato yield, sunlight, soil quality and the tomato variety are extraneous variables. A well-designed experiment handles them with four principles.
Comparison. The experiment has at least two treatment groups, so that the response to the treatment of interest is measured against something. Often one group is a control group, a group created for comparison. It might get no treatment, an inactive imitation of the treatment (a placebo), or the current standard treatment, which is the ethical choice when withholding treatment would harm people. Without comparison, any change you see could come from something else that changed at the same time: tomatoes grow more in a sunny summer whether or not they are fertilized.
Random assignment. Treatments are assigned to the experimental units by a chance process. The purpose is to make the treatment groups as alike as possible on every extraneous variable, including ones nobody thought to measure. If the randomization works, the distribution of soil quality, of sunlight, of anything, is roughly the same in every group, so a difference in the response that is too big to be chance can be put down to the treatments.
The alternative shows why this matters. If the gardener puts the fertilizer on the plots she thinks will grow best, plot quality becomes a confounding variable: a variable linked to which treatment a unit got, so that you can't tell whether the fertilizer or the better plots caused the higher yield. Random assignment breaks that link.
To describe random assignment for 40 plots and two treatments: number the plots 1 to 40, use a random number generator to pick 20 different integers from 1 to 40 (ignoring repeats), give fertilizer to those 20 plots, and leave the other 20 unfertilized. Slips of paper work too: 20 marked "F" and 20 marked "N", mixed in a container, one drawn for each plot.
Replication. More than one experimental unit gets each treatment. Units differ naturally, so one fertilized plot against one unfertilized plot can't separate the fertilizer's effect from the difference between those two plots. With 20 plots per treatment, the natural differences average out within each group, and a real treatment effect shows up as a difference between the group averages.
Direct control. The researcher holds extraneous variables at the same setting for every unit: the same tomato variety, the same watering schedule, the same harvest date. This does two jobs. It removes those variables as possible confounders, and it reduces the variation in the response, which makes a treatment effect easier to see.
Placebos and blinding
People who believe they are being treated often respond, even when the treatment is inactive. The placebo effect is the difference between the average response to a placebo and the average response to no treatment at all. A pain-relief study that only compares a new drug with "nothing" can't tell how much of the drug group's improvement came from the drug and how much from expecting relief. Comparing the drug with a placebo that looks, tastes and is taken exactly like it isolates the drug's own effect.
Expectations affect the people running the study too. A researcher who knows which patients got the drug may rate their symptoms as milder, or treat them a little differently. Blinding (also called masking) prevents this:
- In a single-blind experiment, one side knows and the other does not: usually the participants don't know which treatment they are getting while the researchers who work with them do, though the reverse also counts.
- In a double-blind experiment, neither the participants nor the members of the research team who interact with them or evaluate them know which treatment each participant received. Someone outside that team holds the assignment list.
Blinding isn't always possible. Participants in a yoga program know they are doing yoga. Then the experiment can still blind the people who measure the response, such as the nurse taking blood pressure readings.
Three designs
A completely randomized design assigns the treatments to all the experimental units completely at random, as in the 40 fertilizer plots above. The groups are often the same size, but they don't have to be.
A randomized block design is used when a known extraneous variable has a large effect on the response. That variable is the blocking variable. First, group the units into blocks of units that are alike on the blocking variable. Then randomly assign the treatments separately within each block, so that every treatment appears in every block. If half the tomato plots are in full sun and half are shaded, form a sun block and a shade block, and within each block randomly assign fertilizer to half the plots.
Blocking makes comparisons more precise. In a completely randomized design, the sun-versus-shade difference is part of the random noise in the response, and it can hide a modest fertilizer effect. In a block design, the treatments are compared inside each block, among units that share the same light, so variation caused by sunlight is separated from the comparison. Blocking in an experiment plays the same role as stratifying in a sample (Topic 1.11): group by a variable that matters, then use chance within groups.
A matched pairs design is a randomized block design with two treatments and blocks of size two. Either units are paired by similarity on extraneous variables and one member of each pair is randomly given each treatment, or each unit receives both treatments in a randomly decided order (or, as below, on randomly decided sides).
Here is matched pairs at work. A shoe company compares two sole materials, A and B. Five runners each wear one shoe of each material, with a coin flip deciding which foot gets A, and the sole wear (in millimeters) is measured after 500 km:
| Runner | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| Material A | 2.1 | 3.4 | 5.0 | 2.9 | 4.4 |
| Material B | 2.5 | 3.7 | 5.2 | 3.3 | 4.7 |
| B − A | 0.4 | 0.3 | 0.2 | 0.4 | 0.3 |
The wear differs hugely from runner to runner (from 2.1 mm to 5.2 mm, depending on weight, stride and route), but within every runner B wore 0.2 to 0.4 mm more, a mean difference of 0.32 mm. If ten runners had been split into an A group and a B group, a 0.32 mm difference would be buried under runner-to-runner differences with a standard deviation of over 1 mm. Pairing removed that variation from the comparison. The coin flip matters too: if every runner wore A on the right foot, foot (and its slightly different strike) would be confounded with material.
The right design depends on the study's goals, the units, and the variables involved. Use a completely randomized design when the units are fairly alike or no single extraneous variable stands out. Block when one extraneous variable is known to affect the response strongly and you can measure it before assigning treatments. Use matched pairs when there are two treatments and units can be paired closely or can receive both treatments without one affecting the other (a drug whose effect lingers makes "both treatments in random order" a bad idea).
What an experiment lets you conclude
Because random assignment makes the groups roughly alike on everything but the treatment, with only chance differences left, it reduces the potential for confounding, and a well-designed experiment that finds a clear difference in the response supports a cause-and-effect conclusion: the treatment caused the difference. An observational study (Topic 1.10) can't support that conclusion, because its groups chose themselves.
Who the conclusion applies to is a separate question, answered by how the units were obtained. Topic 1.10 showed that random selection is what lets results generalize to a population. Most experiments on people can't randomly select from a population: participants must consent, so the units are volunteers. The results then apply to units like the ones in the study.
| Treatments randomly assigned | Not randomly assigned | |
|---|---|---|
| Units randomly selected | Cause and effect, for the whole population | Association only, for the whole population |
| Not randomly selected (e.g., volunteers) | Cause and effect, for units like these | Association only, for units like these |
Worked examples
A food scientist studies how storage temperature (2°C or 6°C) and packaging (sealed tray or open tray) affect how long strawberries last. She takes 48 identical containers of strawberries, all from one farm and picked the same morning, and randomly assigns 12 containers to each combination of temperature and packaging. She records the number of days until mold appears in each container. Identify the explanatory variables, the treatments, the experimental units and the response variable, and say how the design uses each of the four principles.
Show solution
Read the study for what the researcher imposes (explanatory variables) and what she measures (response).
- Explanatory variables: storage temperature and packaging. With two factors, each treatment is a combination: 2°C sealed, 2°C open, 6°C sealed, 6°C open, so there are treatments.
- Experimental units: the 48 containers of strawberries.
- Response variable: days until mold appears.
- Comparison: four treatment groups are compared.
- Random assignment: containers are randomly assigned to the treatments, so fruit quality differences should be spread evenly across groups.
- Replication: containers per treatment.
- Direct control: using one farm, one picking morning and identical containers keeps the source of the fruit, its age and the container the same for every unit. None of these can then confound the comparison, and the response varies less, so a treatment effect is easier to see.
A clinic tests a new migraine medication. Ninety volunteers are randomly assigned, 30 each, to take the new pill, take an identical-looking sugar pill, or receive no pill. Each rates pain reduction two hours after a migraine starts, on a 0 to 10 scale. The mean reductions are 3.1 for the new pill, 1.8 for the sugar pill, and 0.6 for no pill.
(a) Estimate the placebo effect, and estimate the new pill's effect beyond the placebo effect.
(b) Can this experiment be double-blind? Explain.
Show solution
(a) The placebo effect is the mean response to the placebo minus the mean response to no treatment: points. The new pill's effect beyond expectation is the comparison with the placebo, not with nothing: points. Comparing the pill with no pill ( points) would credit the pill with the placebo effect too.
(b) Not completely. Patients in the no-pill group know they got nothing, so that group can't be blinded. The pill and sugar-pill groups can be double-blind: the pills look alike, and neither the patients nor the staff who interact with them or record the ratings would know which pill each patient received, with the assignment list kept by someone outside the team.
An agricultural researcher wants to compare three irrigation schedules (I, II and III) on the yield of soybeans. She has 24 plots at a research farm: 12 on a hillside, where water drains quickly, and 12 in a valley, where the soil holds water.
(a) Describe a randomized block design for this experiment.
(b) Explain why a randomized block design is a better choice here than a completely randomized design.
(c) Suppose the schedule II plots produce a much higher mean yield than the others, a difference too large to be explained by chance. What conclusion can the researcher draw, and to which fields does it apply?
Show solution
(a) Block by location: the 12 hillside plots form one block and the 12 valley plots form the other. Within the hillside block, number the plots 1 to 12 and use a random number generator to choose 4 different numbers; those plots get schedule I. Choose 4 different numbers from the remaining 8 for schedule II, and the last 4 plots get schedule III. Repeat the same process, separately, in the valley block. Compare the mean yields of the three schedules within each block.
(b) How fast the soil drains affects how much water the soybean roots get, so location is expected to affect yield strongly, apart from the schedule. In a completely randomized design, chance could put more hillside plots on one schedule, and the large hillside-valley difference would be mixed into the comparison of schedules. Blocking compares the schedules among plots with the same drainage, which separates the variation due to location from the comparison and gives a more precise comparison of the irrigation schedules.
(c) Because the schedules were randomly assigned to plots, the researcher can conclude that schedule II causes higher soybean yield than schedules I and III on these plots. The plots were not randomly selected from all soybean fields, so the conclusion applies only to fields similar to the research farm's hillside and valley plots, not to soybean fields in general.
Where students lose points
On the exam
Free-response Question 1 is built mainly on Practice 1 (formulate questions) and Practice 2 (collect data), and Question 4 assesses Practice 2 along with Practices 3 and 4, so expect to describe a randomization, a block or matched pairs design, or the scope of a conclusion in words, with no calculation. A complete block-design description names the blocks and why the blocking variable matters for the response, and randomizes within every block. A conclusion about cause and effect or about generalization must state the reason (random assignment, or random selection) and be in context.
Check yourself
Question 1
In an experiment comparing two study apps, 60 students are randomly assigned to use one app or the other for a month. What is the main purpose of the random assignment?
- A
To make the results apply to all students in the country
- B
To make the two groups similar on extraneous variables such as prior grades and motivation, so a difference in scores can be attributed to the app
- C
To guarantee that each group has exactly 30 students
- D
To keep the students from knowing which app they are using
Show answer
B
Random assignment balances extraneous variables across the groups, which is what allows a cause-and-effect conclusion. Choice A describes random selection, which these 60 students did not have.
- A
Question 2
A dermatologist compares two hand creams for dry skin on 20 volunteers whose skin dryness varies a lot from person to person. Which design best controls for those person-to-person differences?
- A
Randomly assign 10 volunteers to each cream
- B
Let each volunteer choose the cream they prefer
- C
Have each volunteer use cream A on one hand and cream B on the other, with a coin flip deciding which hand gets A
- D
Give cream A to the 10 volunteers with the driest skin and cream B to the rest
Show answer
C
Each volunteer serves as a pair: both creams are compared on the same skin, so person-to-person differences drop out, and the coin flip keeps hand from being confounded with cream. Choice D confounds dryness with cream, and choice A leaves person-to-person variation inside the comparison.
- A
Question 3
In a test of a new flu nasal spray, patients do not know whether their spray is the real one or a saltwater spray, but the nurses who give the sprays and record symptoms do know. Is this experiment single-blind or double-blind? What risk does that leave?
Show answer
Single-blind. The nurses' knowledge could bias how they record symptoms or how they treat patients, in favor of the new spray.
Double-blind requires that neither the patients nor the research staff who interact with them know the assignments. Here only the patients are blinded, so the staff's expectations can still push the recorded response in one direction.
Question 4
A researcher recruits 150 volunteers at one city's gyms and randomly assigns them to a high-protein or standard diet for 8 weeks. The high-protein group gains much more muscle mass, a difference too large to be due to chance. What can she conclude, and for whom?
Show answer
The high-protein diet caused the greater muscle gain, but only for people similar to these volunteers (gym-goers in that city who would volunteer), not for adults in general.
Random assignment supports cause and effect. The volunteers were not randomly selected from any larger population, so the conclusion extends only to people like them.