The Reality of Imperfect Experiments
In a perfect world, everyone in an experiment follows instructions. In reality, they don't. This gap between assignment and action is called **noncompliance**, and it's a core challenge in field experiments. This guide explores how to draw valid conclusions when faced with the common scenario of one-sided noncompliance.
Who Are the People in Our Study?
Noncompliance isn't random. To understand its effects, we first classify people into hidden groups based on how they would react to any assignment. This is called **Principal Stratification**.
With one-sided noncompliance, the control group cannot access the treatment. This structurally eliminates Always-Takers and Defiers.
Compliers
Take the treatment if and only if they are assigned to it. Their behavior follows the assignment.
Never-Takers
Will not take the treatment, regardless of their assignment.
Always-Takers
Will always take the treatment, regardless of their assignment.
Defiers
Do the opposite of their assignment. They are generally assumed not to exist.
The Analytical Toolkit
To estimate a causal effect amidst noncompliance, we need a clear set of assumptions and a precise definition of what we are trying to measure.
Non-Interference (SUTVA)
An individual's outcome is not affected by anyone else's treatment assignment. (No spillover effects).
Exclusion Restriction
The *assignment* to treatment only affects the outcome through its effect on *receiving* the treatment. Mere assignment (e.g., getting an encouragement letter) has no direct effect on Never-Takers.
Monotonicity
There are no "Defiers." The assignment to treatment can't make someone *less* likely to take it. This is automatically satisfied by design in one-sided noncompliance.
Intention-to-Treat (ITT)
The effect of *assigning* treatment, regardless of who takes it up. Answers a policy question: "What is the effect of offering the program?" It's a simple comparison of the two randomized groups.
Complier Average Causal Effect (CACE)
The effect of *receiving* treatment, but only for the group of Compliers. Answers a scientific question: "For those who took the treatment because we offered it, what was the effect?" Also known as the Local Average Treatment Effect (LATE).
Average Treatment Effect (ATE)
The average effect if everyone in the population received the treatment. This is the ideal but is **not identified** with noncompliance because we never learn the treatment effect for Never-Takers.
Interactive Solution: The Wald Estimator
Let's make this concrete with data from the Sommer & Zeger (1991) study on Vitamin A supplements and child mortality. The goal is to find the causal effect of the Vitamin A supplement itself (the CACE).
Study Data
The CACE Formula (Wald Estimator)
The CACE is simply the Intention-to-Treat effect on the outcome (ITT_Y) divided by the Intention-to-Treat effect on treatment take-up (ITT_W), which is the compliance rate.
Calculated Effects
Mortality Rate by Assignment (ITT)
Treatment Take-up Rate (Compliance)
Common Analytical Pitfalls
Intuitive approaches to handling noncompliance can lead to severely biased results. It's crucial to avoid these common mistakes.
DON'T: As-Treated or Per-Protocol Analysis
This involves ignoring the original random assignment and either comparing everyone who got the treatment to everyone who didn't, or dropping non-compliers.
Why it's wrong: This breaks the randomization! The groups are no longer comparable because the reasons people choose to comply (or not) are often related to the outcome. This introduces **selection bias**. A famous example is the "healthy adherer effect," where people who stick to a placebo regimen have better outcomes, not because the placebo works, but because they are systematically healthier or more conscientious.
DO: Intention-to-Treat (ITT) & CACE
Always start with the ITT analysis, which respects the original randomization. It gives an unbiased estimate of the effect of the *policy* or *offer*.
Then, if the assumptions hold (especially the exclusion restriction), use an instrumental variable approach (like the Wald estimator shown above) to estimate the CACE. This gives an unbiased estimate of the effect of the *treatment* for the subpopulation of Compliers.
Design in Anticipation of Noncompliance
The best analysis can't fix a poorly designed experiment. Proactive design is key to maximizing both compliance and the credibility of your results.
Strategies to Maximize Compliance
- Simplify the Intervention: Make participation as easy and low-cost as possible.
- Use Behavioral Nudges: Employ reminders, clear messaging, and effective framing to encourage take-up.
- Educate and Communicate: Ensure participants understand the protocol and its benefits. Build trust.
Designing for a Credible Analysis
- Strengthen the Exclusion Restriction: Use placebos and double-blinding where possible to ensure the assignment itself doesn't have a direct effect.
- Use Encouragement Designs Carefully: If using an encouragement as an instrument, ensure it doesn't provide extra information that could independently affect the outcome.
- Measure Key Covariates: Collect baseline data on variables that might predict compliance and outcomes to improve precision and characterize the Complier population.