The Achilles' Heel of Field Experiments

Attrition, the loss of participants over time, is one of the most persistent threats to the validity of randomized controlled trials. It undermines the very foundation of randomization—that the treatment and control groups are comparable—and can reintroduce the selection bias the experiment was designed to eliminate.

Diagnosing Attrition Bias

The core problem is **differential attrition**, where the rate or nature of dropouts differs between groups. This breaks the baseline equivalence randomization created. Click the button below to see how differential attrition can unbalance baseline characteristics in the final analyzed sample.

Scenario: No Attrition Bias (Balanced Attrition)

Attrition Rates

Baseline Covariate Balance

With no attrition bias, both attrition rates and baseline characteristics of the remaining subjects are balanced across groups. The comparison remains valid.

Placing Bounds on the Treatment Effect

When attrition is a problem, we can't be certain about the true Average Treatment Effect (ATE). Bounding methods make this uncertainty explicit by calculating a range of plausible values for the ATE based on different assumptions about the missing data.

Interactive Bounding Calculator

Use the sliders to see how attrition rates affect the estimated bounds on the ATE. We assume the true outcome for respondents is 60 in the treatment group and 40 in the control group, with an outcome range of 0 to 100.

Manski "Worst-Case" Bounds

Assumption-free. Imputes extreme values for attritors.

Lee "Trimming" Bounds

Assumes treatment only increases response (monotonicity).

Visualizing Lee Bounds

To get the bounds, we "trim" the outcomes from the group with lower attrition (higher response rate) to make it comparable to the other group.

Analytical Strategies for Correction

When we are willing to make stronger assumptions, we can use statistical models to generate a single point estimate of the treatment effect, corrected for attrition bias.

Inverse Probability Weighting (IPW)

IPW works by giving more weight to observed individuals who are similar to those who dropped out. This creates a re-weighted sample that statistically mimics the original, full sample.

Key Assumption: Missing at Random (MAR)Attrition can depend on observed baseline characteristics, but not on the unobserved outcomes themselves, once we control for those characteristics.

Heckman Selection Models

The Heckman model directly addresses selection bias by modeling the attrition process itself. It's designed for cases where attrition is thought to depend on unobserved factors.

Key Assumption: Exclusion RestrictionYou must have an "instrument"—a variable that predicts attrition but has no direct effect on the outcome of interest. This is a very strong and hard-to-satisfy assumption.

Multiple Imputation (MI)

Instead of re-weighting or discarding data, MI "fills in" the missing values multiple times, creating several complete datasets. The analysis is run on each dataset, and the results are pooled to produce a final estimate that properly accounts for the uncertainty of the missing data.

Key Assumption: Missing at Random (MAR)Like IPW, MI assumes that we have enough observed data to reliably predict the values of the missing data.

Prevention is Better Than Cure

While analytical corrections are valuable, the best strategy is to prevent attrition in the first place. Good design and thoughtful fieldwork are a researcher's most powerful tools.

A Researcher's Prevention Checklist

Tracking & Engagement

  • Collect comprehensive tracking info at baseline (multiple numbers, emails, secondary contacts).
  • Maintain regular, positive contact (newsletters, holiday cards).
  • Minimize participant burden with concise, well-piloted surveys.

Incentive Design

  • Use cash or cash-equivalent incentives (e.g., gift vouchers).
  • Employ unconditional incentives (upfront) to build reciprocity.
  • For longitudinal studies, use a phased or escalating incentive structure with a completion bonus.

Additional Data Collection

  • Plan for **refreshment samples**: new random samples drawn during follow-up to diagnose the nature of attrition bias.
  • Conduct intensive follow-up on a random sub-sample of attritors to get better data for weighting or bounds.