Complete lesson: observational studies vs experiments, correlation vs causation, bias, generalizability, and 10 practice questions.
📖 What Are Statistical Claims?
Every day, we see statistics in the news, in advertisements, and in research. Evaluating statistical claims means analyzing whether the conclusions drawn from data are valid and well-supported.
📌 Key Concept
Two important questions to ask: 1. Who was studied? (Can we generalize the results?) 2. How was the study conducted? (Can we conclude causation?)
Observational Study
Researchers observe and measure without interfering. Can show correlation, NOT causation.
Experiment
Researchers assign treatments to subjects. Can show causation.
Correlation
Two variables are related. Does NOT mean one causes the other.
Causation
One variable directly causes a change in another. Requires an experiment.
1. Observational Studies vs Experiments
📌 RULE: Types of Studies
Observational Study:
• Researchers observe subjects without intervention.
• Can identify correlations between variables.
• Cannot prove causation.
Experiment:
• Researchers assign treatments to subjects.
• Can identify causation.
• Random assignment helps control for confounding variables.
🚨 CRITICAL RULE
Correlation does NOT imply causation!
Just because two things are related does not mean one causes the other.
📝 SOLVED EXAMPLE 1 — Observational Study
A study found that people who drink coffee have a lower risk of heart disease. Is this evidence that coffee prevents heart disease?
Step 1: This is an observational study (no treatment assigned).
Step 2: Observational studies can only show correlation.
Step 3: Coffee drinkers may have other habits that reduce heart disease risk.
Step 4:No, this does not prove causation.
✅ No — this is correlation, not causation.
💡 Tip: To prove causation, you need a controlled experiment with random assignment.
2. Correlation vs Causation
📌 RULE: Correlation vs Causation
Correlation: Two variables move together (positive or negative).
Causation: One variable directly causes a change in the other.
Example: Ice cream sales and drownings are correlated (both increase in summer). But ice cream doesn't cause drowning — both are caused by hot weather (a confounding variable).
💡 Strategy — Evaluating Claims
1. Identify if the study is observational or experimental.
2. If observational → correlation only.
3. If experimental with random assignment → causation possible.
4. Check for bias in sample selection.
📝 SOLVED EXAMPLE 2 — Correlation vs Causation
A study shows that students who eat breakfast have higher test scores. Can we conclude that eating breakfast causes higher test scores?
Step 1: This is likely an observational study.
Step 2: Students who eat breakfast may also study more or have better sleep habits.
Step 3:No, we cannot conclude causation.
✅ No — correlation only.
💡 Tip: Always look for other possible explanations (confounding variables).
3. Sampling and Bias
📌 RULE: Good vs Bad Sampling
Good:
• Random sample: Everyone in the population has an equal chance of being selected.
• Representative sample: The sample reflects the population.
Bad (Bias):
• Voluntary response: Only people who choose to respond.
• Convenience sample: Only people who are easy to reach.
• Undercoverage: Some groups are left out.
📌 Key Concept
A random sample allows us to generalize results to the entire population. A biased sample does not.
📝 SOLVED EXAMPLE 3 — Sampling Bias
A survey about school lunch quality asks only students who eat in the cafeteria. Is this a representative sample?
Step 1: Students who don't eat in the cafeteria are excluded.
Step 2: This is a biased sample (undercoverage).
Step 3: Results cannot be generalized to all students.
✅ No — this is biased (undercoverage).
💡 Tip: A good sample represents the entire population.
4. Generalizing Results
📌 RULE: Generalizability
Results can only be generalized to the population from which the sample was drawn.
Example: A survey of students at a single high school cannot be generalized to all high school students in the country.
📝 SOLVED EXAMPLE 4 — Generalizability
A study of 500 randomly selected adults in New York City finds that 60% support a new law. Can we conclude that 60% of all Americans support the law?
Step 1: The sample is from New York City only.
Step 2: New York City may not represent all of America.
Step 3:No, we can only generalize to New York City adults.
✅ No — only to New York City adults.
💡 Tip: You can only generalize to the population that was actually sampled.
🧪 Practice Questions
Solve each problem using the rules above. Click "Show Answer" to see the full solution.
Question 1
A study finds that people who exercise more tend to weigh less. What can we conclude?
A) Exercise causes weight loss
B) There is a correlation between exercise and weight
C) Exercise always leads to weight loss
D) Weight causes people to exercise
✓ Answer: B
The study shows a relationship (correlation), but without an experiment, we cannot conclude causation.
📝 Solution: Observational study → correlation only.
Question 2
A researcher wants to test if a new drug lowers blood pressure. What type of study should they conduct?
A) Observational study
B) Survey
C) Controlled experiment with random assignment
D) Case study
✓ Answer: C
To prove causation (the drug lowers blood pressure), you need an experiment with random assignment to treatment and control groups.
📝 Solution: Experiment needed for causation.
Question 3
A survey asks people to call a phone number to share their opinion. What type of sample is this?
A) Random sample
B) Voluntary response sample
C) Representative sample
D) Simple random sample
✓ Answer: B
People who call in are choosing to respond (voluntary response). This is biased because only people with strong opinions will call.
📝 Solution: Voluntary response → biased.
Question 4
A study of 100 randomly selected students at a university found that 70% prefer online classes. Which statement is correct?
A) 70% of all students in the country prefer online classes
B) 70% of all students at this university prefer online classes
C) Approximately 70% of students at this university prefer online classes
D) Exactly 70% of students in the world prefer online classes
✓ Answer: C
The sample is from the university, so results can only be generalized to that university. Also, the 70% is an estimate.
📝 Solution: Generalize to the sampled population only.
Question 5
Which type of study can prove causation?
A) Observational study
B) Survey
C) Case study
D) Controlled experiment
✓ Answer: D
Only a controlled experiment with random assignment can prove causation.
📝 Solution: Experiment → causation possible.
Question 6
A study found that ice cream sales and shark attacks are correlated. What is the likely explanation?
A) Ice cream causes shark attacks
B) Both are caused by summer weather (confounding variable)
C) Sharks attack people who eat ice cream
D) Ice cream companies pay for shark attacks
✓ Answer: B
Both variables are related to summer (more people eat ice cream and swim in the ocean). This is a confounding variable.
📝 Solution: Confounding variable → no causation.
Question 7
A researcher randomly assigns students to either a tutoring program or no tutoring. Both groups take a test. This is an example of:
A) Observational study
B) Correlational study
C) Controlled experiment
D) Survey
✓ Answer: C
Random assignment to treatment groups makes this a controlled experiment.
📝 Solution: Random assignment → experiment.
Question 8
A survey asks only students in the honors program about their study habits. What is this an example of?
A) Bias (undercoverage)
B) Random sampling
C) Experiment
D) Causation
✓ Answer: A
Honors students may not represent all students. This is undercoverage bias.
📝 Solution: Undercoverage → biased sample.
Question 9
A study of 1,000 randomly selected adults in Texas found that 55% support a new law. Can this be generalized to all U.S. adults?
A) Yes, because it's a large sample
B) No, because the sample is only from Texas
C) Yes, because it's random
D) No, because the sample is too small
✓ Answer: B
The sample was drawn from Texas, so results can only be generalized to Texas adults.
📝 Solution: Only generalize to the sampled population.
Question 10
A study finds that students who sleep more get better grades. Which statement is correct?
A) There is a correlation between sleep and grades
B) Sleeping more causes better grades
C) All students who sleep more will get better grades
D) Better grades cause more sleep
✓ Answer: A
Observational studies show correlation, not causation. There could be other factors (like motivation).
📝 Solution: Correlation only.
🎉 WELL DONE!
You've completed the Evaluating Statistical Claims lesson. You now know how to distinguish between observational studies and experiments, identify correlation vs causation, recognize bias in sampling, and evaluate whether results can be generalized.