"Does a new reading app help fourth graders?" sounds like a question, but it doesn't tell you what to measure, how to analyze it, or whom the answer applies to. Topic 1.1 said an investigative question is fixed before any data are collected. A complete one also names the variables, the analysis and the conclusion it is after, and the way the data were collected decides which conclusions the question can actually reach.
The three parts of an investigative question
A well-built investigative question does three jobs.
1. It names the variables. The question is phrased in terms of the variable or variables you will record, so it tells you what data to collect. "Does the app help?" doesn't; "Does using the app for 8 weeks change a student's score on the district reading test?" names an explanatory variable (app or no app) and a response variable (the change in test score).
2. It points to the analysis. It makes clear which parameter the study is about. A parameter is a number that describes a whole population, such as the proportion of all district voters who support a bond, or the mean commute time of all of a city's workers. (Units 3 and 4 develop inference for proportions and means.) Then:
- If the plan is a hypothesis test, the question shows the direction of the claim being tested: greater than, less than, different from, or, for two categorical variables, whether they are associated (not independent). "Is the mean improvement greater for students who use the app?" points to a one-sided test.
- If the plan is a confidence interval, the question asks to estimate the parameter's value as a range of plausible values: "What is the mean improvement in reading score for students who use the app?"
3. It says what kind of conclusion is possible. It names the population the results will apply to and, when the study is an experiment with random assignment, it asks a cause-and-effect question ("Does using the app cause higher scores...?"). Which population and whether causation is allowed both depend on how the data are collected: whether individuals were randomly selected, and whether treatments were randomly assigned.
A census
A census collects data from every individual in the population. A teacher who records the commute time of all 28 students in her class, to describe that class, has taken a census; so does the U.S. Census Bureau every ten years, attempting to count every resident of the country. With a census, a parameter is calculated directly rather than estimated from a sample, so there is no sampling variability to account for. Most populations are too large, too spread out or too expensive to measure completely, which is why most studies use samples (Topic 1.1).
Experiments
In an experiment, the researcher assigns conditions, called treatments, to the individuals being studied and then measures what happens. The vocabulary is precise, and exam questions ask you to identify each piece:
- Experimental units are the things treatments are assigned to: plants, batteries, classrooms, people. When the units are people they are often called subjects or participants.
- An explanatory variable, also called a factor, is a variable the researcher sets for each unit. Its possible values are its levels.
- With one factor, the treatments are its levels. With more than one factor, each treatment is a combination of levels, one from each factor, so the number of treatments is the product of the numbers of levels: 3 doses of a drug and 2 dosing schedules give treatments.
- The response variable is the outcome measured on each unit after its treatment.
The defining feature is that the researcher imposes the treatment. Topic 1.13 covers what makes an experiment well designed (comparison, random assignment, replication and control).
Observational studies and confounding
In an observational study, the researcher imposes nothing: they record the variables of interest as they already are. Three kinds come up:
- A prospective study selects its individuals now and gathers data both now and into the future (enroll 2,000 adults today, record their diets, check their health every year for ten years).
- A retrospective study selects its individuals now and gathers data about their past (pick 500 patients with a disease and 500 without, and look up their medical records from the previous decade).
- A survey collects data from people by asking everyone the same set of questions.
The weakness of every observational study is confounding. A confounding variable offers another explanation for an association between the explanatory and response variables. To confound, a variable has to be associated with both. Suppose a study finds that coffee drinkers have more heart disease. Smoking could confound this: in many populations heavy coffee drinkers are more likely to smoke (associated with the explanatory variable), and smoking causes heart disease (associated with the response). The study can't tell whether coffee, smoking or both are responsible. A variable linked to only one of the two, like a family history of heart disease that has nothing to do with coffee habits, can't explain the association away.
In an experiment, random assignment to treatments spreads variables like smoking roughly evenly across the groups, which is why experiments can support causal conclusions and observational studies can't.
What a study lets you conclude
Two questions about how the data were collected settle the scope of the conclusion, and they are separate questions.
How were the individuals selected? A sample is random when every individual in it was chosen by a chance mechanism, such as a random number generator picking names from a list of the population (Topic 1.11 covers the methods). Then the results can be generalized to the whole population that was sampled. When individuals were picked deliberately by the researcher, or volunteered themselves, the results apply only to individuals like the ones in the study. A survey of people who chose to answer an online poll speaks for people who answer online polls.
How were treatments assigned? If the researcher randomly assigned treatments, a difference in the response that is too large to be chance can be attributed to the treatments: a cause-and-effect conclusion. Without random assignment, including every observational study, only an association can be concluded.
An experiment on volunteers can show causation but only for people like the volunteers; a random-sample survey can describe the whole population but can't show causation. A drug trial that enrolls volunteer patients is the first kind: it can show the drug works, but only for patients like the ones who enrolled.
Worked examples
A horticulturist has 60 tomato plants. She randomly assigns each plant to one of three amounts of fertilizer (none, 5 grams per week, 10 grams per week) and one of two watering schedules (daily, every other day), with 10 plants per combination. After 12 weeks she weighs the tomatoes from each plant. Identify the experimental units, the factors and their levels, the treatments, and the response variable.
Show solution
The researcher assigns the fertilizer amounts and watering schedules, so this is an experiment.
- Experimental units: the 60 tomato plants, the things the conditions are assigned to.
- Factors: fertilizer amount, with 3 levels (0, 5 and 10 grams per week), and watering schedule, with 2 levels (daily, every other day).
- Treatments: every combination of one level of each factor, treatments: no fertilizer with daily watering, no fertilizer with every-other-day watering, 5 grams with daily, 5 grams with every other day, 10 grams with daily, 10 grams with every other day. With 10 plants per treatment, plants are used.
- Response variable: the total weight of tomatoes produced by each plant after 12 weeks.
Researchers reviewed five years of health records for 1,200 office workers and found that those who usually took the stairs had lower average blood pressure than those who usually took the elevator. (a) What kind of study is this? (b) Identify a possible confounding variable and explain how it could account for the association.
Show solution
(a) The researchers did not assign anyone to take the stairs or the elevator; they recorded what workers already did, so this is an observational study. They selected the workers now and gathered data from the past five years of records, so it is retrospective.
(b) Overall fitness is a possible confounding variable. Fitter workers are more likely to choose the stairs, because climbing them is easier for them (associated with the explanatory variable, stairs or elevator). Fitter people also tend to have lower blood pressure, whatever they do at work (associated with the response variable). So the stair-takers might have lower blood pressure because they are fitter, not because of the stairs. Each link has to be stated: a variable connected to only one of the two can't produce the association.
A school district has 40 fourth-grade classrooms. The superintendent asked for teachers willing to try a reading app, and 12 volunteered. Six of the 12 classrooms were chosen at random to use the app for 8 weeks, and the other 6 did not use it. For each classroom, the district recorded the mean change in students' reading test scores over the 8 weeks.
(a) Identify the experimental units, the explanatory variable and the response variable. (b) Can the district conclude that the app causes improvement? To which classrooms do the conclusions apply? (c) Write an investigative question for a hypothesis test about the app that includes all three components.
Show solution
(a) The experimental units are the 12 classrooms, since the app was assigned to whole classrooms. The explanatory variable is whether the classroom used the app (two treatments: app, no app). The response variable is the classroom's mean change in reading test score over the 8 weeks.
(b) The app was randomly assigned to 6 of the 12 classrooms, so if the app classrooms improve by convincingly more, the district can conclude that the app caused the greater improvement. But the 12 classrooms were volunteers, not a random sample of the district's 40 fourth-grade classrooms. Teachers who volunteer to try new tools may differ from those who don't, so the conclusions apply only to classrooms like those whose teachers volunteered, not to all 40.
(c) "For fourth-grade classrooms like those whose teachers volunteered in this district, does using the reading app for 8 weeks cause a greater mean improvement in reading test scores than not using it?"
- Variables: app use (yes or no) and the change in reading test score.
- Parameter and direction: the difference between the mean improvement with the app and without it, with the alternative that it is greater with the app.
- Scope: classrooms like the volunteers' (no random sample), with a causal claim (random assignment).
Where students lose points
On the exam
Questions on this topic can be conceptual and can sit inside a longer free-response question about a study: identify the type of study, its units and variables, a confounding variable with both links explained, or the scope of the conclusions, each in context. An investigative question you write is checked for the three components: the variables, the parameter with its direction or estimation goal, and the population with any causal claim.
Check yourself
Question 1
Which of the following is a census?
- A
A city surveys 400 randomly chosen households about recycling habits to estimate the habits of all households in the city
- B
A coach records the resting heart rate of every player on her 18-player team to describe the team
- C
A website asks visitors to rate a new film and reports the average of 5,000 ratings
- D
A factory tests every 50th battery off the line to monitor the quality of its batteries
Show answer
B
The population is the coach's team, and she records every member of it. A and D are samples (a random sample and a systematic one), and C is a volunteer sample of the film's viewers.
- A
Question 2
A baker tests 4 oven temperatures and 3 baking times on loaves of bread, assigning loaves at random so that every combination is used. How many treatments are there?
- A
3
- B
4
- C
7
- D
12
Show answer
D
With two factors, each treatment is one temperature combined with one time, so there are treatments. Adding the levels (C) counts the levels, not the combinations; A and B count only one factor's levels.
- A
Question 3
An observational study of high school students finds that those who eat breakfast every day have higher GPAs, on average, than those who don't. Identify a possible confounding variable and explain how it could account for the association.
Show answer
Amount of sleep. Students who get more sleep are more likely to wake up early enough to eat breakfast, and well-rested students also tend to concentrate better and earn higher grades. So breakfast eaters might have higher GPAs because they sleep more, not because of breakfast.
A confounding variable must be associated with both the explanatory variable (eating breakfast) and the response (GPA). Other variables work too, such as family routines or how organized a student is, as long as both links are explained.
Question 4
To study whether screen time is related to eyesight, researchers enroll 800 nine-year-olds this year, measure their eyesight and their daily screen time, and plan to measure both again every year until the children are 18. What kind of study is this?
- A
A prospective observational study
- B
A retrospective observational study
- C
An experiment, because eyesight is the response variable
- D
A census, because every child's eyesight is measured
Show answer
A
No screen time is assigned, so it is observational, and data are gathered now and into the future, so it is prospective. A retrospective study (B) would look back at past records. Having a response variable doesn't make a study an experiment (C), and 800 children are a sample, not every nine-year-old (D).
- A
Question 5
A pollster randomly selects 1,500 adults from a list of all registered voters in a state and finds that those who listen to talk radio are more likely to oppose a ballot measure. What can the pollster conclude, and what can't she?
Show answer
Because the sample was randomly selected from the state's registered voters, she can generalize the association to all registered voters in the state. Because no one was assigned to listen to talk radio, she can't conclude that listening to talk radio causes opposition to the measure.
Random selection settles who the results apply to; random assignment, which this survey doesn't have, would be needed for cause and effect.