Sampling, Margin of Error, and Evaluating Statistical Claims
This lesson combines two closely related, officially named skills from College Board's Problem-Solving and Data Analysis domain: "Inference from sample statistics and margin of error," and "Evaluating statistical claims: observational studies and experiments." Both are about the same underlying question, how much can you actually trust a conclusion drawn from data? This free, complete lesson covers populations, samples, and random selection, margin of error and estimating population parameters, the crucial difference between observational studies and experiments, and how to evaluate statistical claims, including the classic distinction between correlation and causation. Each idea is followed by a fully worked example, and then a set of original practice problems with instant feedback. Everything here is free, and you can keep practicing afterward with the full SAT Math Question Bank linked below.
1. Populations, Samples, and Random Selection
A population is the entire group a researcher wants to learn about. A sample is the smaller part of the population that's actually studied. Researchers use a sample to make an inference, a generalization, about the whole population. For that inference to be trustworthy, the sample generally needs to be reasonably large and selected using random selection, so that every member of the population has a fair chance of being included.
A critical fact: what matters for reliability is the size of the sample, not the percentage or fraction of the population it represents. A random sample of 500 people is roughly equally reliable whether it's drawn from a population of 50,000 or 5,000,000.
Worked Example: An opinion survey about a new park was conducted in two cities. City A has 200,000 residents and City B has 90,000 residents. Each survey used a random sample of 500 residents from its city. Which statement is true?
Since both samples are the same size and both were randomly selected, they are approximately equally reliable, despite the different population sizes. The size of the population itself doesn't determine reliability, the sample size does.
1. A researcher wants to survey students at a university to gauge opinions on a new policy. Which method would produce the least biased sample?
Explanation: Random selection from the entire student directory gives every student an equal chance of being chosen, avoiding the bias introduced by location, self-selection, or limiting the sample to one group.
2. A poll uses a random sample of 800 people from a state with 2 million residents. Another poll uses a random sample of 800 people from a different state with 8 million residents. Which poll is more reliable?
Explanation: Both samples are the same size (800), and reliability depends on sample size, not on how large the underlying population is.
3. Which sampling method is most likely to introduce bias?
Explanation: Relying on volunteers who choose to respond introduces self-selection bias, since people with strong opinions are more likely to respond than a representative cross-section of the population.
4. As sample size increases (with random selection maintained), what generally happens to the reliability of an estimate?
Explanation: Larger random samples tend to produce estimates closer to the true population value, increasing reliability.
2. Margin of Error
The margin of error describes how far a sample statistic might reasonably differ from the true population value. A result reported as "54% ± 3%" means the true population percentage is plausibly anywhere from 51% to 57%. A larger sample size generally produces a smaller margin of error, a more precise estimate.
Worked Example: A survey of 1,000 voters found that 54% support a candidate, with a margin of error of ±3%. What is the plausible range for the true population percentage?
54% − 3% = 51% 54% + 3% = 57%
The true population percentage is plausibly between 51% and 57%.
1. A poll of 600 people found that 62% favor a new law, with a margin of error of ±4%. What is the range of plausible values for the true population percentage?
Explanation: 62% − 4% = 58% and 62% + 4% = 66%.
2. A survey has a margin of error of ±2.5%. If the sample result is 48%, which of the following is within the plausible range for the true population value?
Explanation: The plausible range is 48% − 2.5% = 45.5% to 48% + 2.5% = 50.5%. Only 49% falls within that range.
3. If a survey's sample size is increased significantly while keeping the same sampling method, what typically happens to the margin of error?
Explanation: Larger samples generally produce smaller margins of error, meaning a more precise estimate of the true population value.
4. A survey of 400 people has a margin of error of ±5%. A similar survey of 1,600 people (4 times the sample size) would most likely have a margin of error that is:
Explanation: Increasing the sample size reduces the margin of error, so the larger survey would have a smaller margin of error than ±5%.
5. A survey estimates that 40% of a population supports a proposal, with a margin of error of ±3.5%. Can we be certain that more than 45% of the population supports the proposal?
Explanation: The plausible range is 40% − 3.5% = 36.5% to 40% + 3.5% = 43.5%. Since 43.5% is still below 45%, we cannot conclude that more than 45% of the population supports the proposal.
3. Observational Studies vs. Experiments
This distinction is one of the most frequently tested ideas in this part of the SAT. In an observational study, researchers measure existing variables without assigning any treatment, so the study can only show an association between variables. In an experiment, researchers actively and randomly assign subjects to different treatment groups, which allows the study to support a causal conclusion.
| What was randomized | What it supports |
|---|---|
| Random selection of the sample from the population | Generalizing results to the whole population |
| Random assignment of subjects to treatment groups | Drawing a cause-and-effect conclusion |
Worked Example: A study observes that people who exercise regularly have lower rates of heart disease, based only on measuring their existing habits, with no assignment to groups. What type of study is this, and what can be concluded?
This is an observational study. It can show that exercise and lower heart disease rates are associated, but it cannot establish that exercise causes the lower rate, since the two groups might differ in other important ways, like diet or age, that weren't controlled for.
1. Researchers randomly assign volunteers to either take a new medication or a placebo, then measure recovery times. What type of study is this?
Explanation: Randomly assigning subjects to a treatment or control group is the defining feature of an experiment.
2. A study finds that students who eat breakfast tend to have higher test scores, based on data collected without any assignment to groups. Can this study establish that eating breakfast causes higher test scores?
Explanation: Without random assignment, this is an observational study, which can only show an association, not a causal relationship, no matter how large the sample is.
3. What is required for a study to support a cause-and-effect conclusion?
Explanation: Random assignment is what allows researchers to rule out other explanations for a difference between groups, making a causal conclusion possible.
4. A researcher wants results that generalize to an entire population. What is required for this?
Explanation: Random selection of the sample is what makes the sample representative of the population, allowing results to be generalized.
5. A study randomly selects participants from a population AND randomly assigns them to treatment groups. What can this study's results support?
Explanation: Random selection supports generalizing to the population, and random assignment supports a causal conclusion. A study with both can support both types of claims.
4. Evaluating Statistical Claims
Being able to spot a flawed statistical claim, or recognize exactly what a well-designed study can and can't support, is its own tested skill. Two ideas come up constantly: a strong association between two variables does not by itself prove one causes the other, and a conclusion should never be extended beyond the population that was actually studied.
1. A news headline states, "Study shows ice cream sales cause increased drowning rates," based on data showing both rise together in the summer. What is the likely flaw in this claim?
Explanation: Ice cream sales and drownings both increase in warm weather, when more people swim and buy ice cream. This shared underlying cause, not a direct causal link between the two, explains the association.
2. A study surveys only adults in one city and concludes the results apply to all adults nationwide. What is the problem with this conclusion?
Explanation: Since only one city's adults were sampled, the results can only be reliably generalized to that city's adult population, not to the entire country.
3. A randomized controlled experiment finds that a new drug significantly reduces symptoms compared to a placebo. Is it reasonable to conclude the drug caused the improvement?
Explanation: A properly randomized experiment with a placebo control group is exactly the kind of design that supports a causal conclusion.
4. Which of the following claims is best supported by an observational study alone?
Explanation: An observational study can only support a claim about association, like choice A. The other three choices all make causal claims, which require an experiment with random assignment.
Common Mistakes to Avoid
- Confusing sample size with population size when judging reliability. A bigger population being sampled from doesn't make a fixed-size sample more or less reliable, only the sample's own size and how it was selected matter.
- Confusing random selection with random assignment. These are two completely different steps that support two completely different kinds of conclusions, generalization versus causation. Mixing them up is the single most common error in this topic.
- Assuming any correlation implies causation. A strong, even statistically significant, association between two variables is never enough on its own to prove one causes the other.
- Overgeneralizing results beyond the sampled population. A study's conclusions only reliably apply to the population it actually sampled from, extending them further is not statistically justified.
- Treating the margin of error as exact bounds guaranteed with certainty. A margin of error gives a plausible range for the true value, it doesn't guarantee the true value falls in that range with 100% certainty.
Frequently Asked Questions
What's the single most important distinction to remember from this lesson?
Random selection versus random assignment. Random selection of who is in the sample lets you generalize results to the broader population. Random assignment of subjects to treatment groups lets you draw causal conclusions. A study can have one, both, or neither, and what it can conclude depends entirely on which it has.
Can an observational study ever be useful if it can't prove causation?
Absolutely. Observational studies are often the only ethical or practical way to study certain questions, and they're extremely valuable for identifying associations worth investigating further, generating hypotheses, and informing decisions, even without proving a causal link.
Why doesn't a large sample size fix the causation problem in an observational study?
A larger sample size makes an association more statistically reliable and precise, but it does nothing to rule out other explanations, like a confounding variable, for that association. Sample size and study design address two completely different sources of uncertainty.
Where can I practice more problems like these?
The Inference from Sample Statistics and Margin of Error quiz and the Evaluating Statistical Claims quiz in the SAT Math Question Bank include additional original problems on these topics, along with quizzes covering every other Algebra, Advanced Math, Problem-Solving and Data Analysis, and Geometry and Trigonometry skill on the Digital SAT.