Understanding Heterogeneous Treatment Effects in Experimental Research
Instead of a simple "Does it work?" (the ATE), experiments often explore the average impact. This can obscure crucial details. Analyzing Heterogeneous Treatment Effects (HTE) provides a richer view, probing: "For whom and why does it work?" This segment illustrates the move from one average result to several, conditional effects.
A solitary average, reflecting population-wide impact. Potent yet potentially deceptive.
Analyzes subgroup effects, identifying those who gain the most or potentially experience negative outcomes.
The MTO study highlights HTE's significance. This program provided housing vouchers to low-income families. While early results showed no overall impact on adult earnings, a subsequent Heterogeneous Treatment Effects (HTE) analysis, detailed below, uncovered crucial differences based on the age of children at the time of relocation.
The data reveals a stark contrast: childhood movers (under 13) experienced a significant 31% income boost, whereas teenage movers saw a slight detriment. This overlooked result, missed by prior analyses, fundamentally reshaped policy conclusions and highlighted the program's unexpected benefits.
Here are a few rewritten options, aiming for a similar length and clarity: **Option 1 (Concise):** > HTE discovery methods have shifted. Historically, it was theory-driven. Now, machine learning unveils hidden patterns. Compare these two influential techniques below. **Option 2 (Slightly more detail):** > How we find HTE has changed. Early approaches confirmed pre-set ideas. Today, machine learning explores data for novel insights. See how these methods differ below. **Option 3 (Emphasis on power):** > HTE's discovery process has transformed. The past relied on set theories; the present uses machine learning. Compare these potent strategies below.
This method begins with a defined hypothesis about a subgroup's unique impact. It's the preferred method for theory testing, often employing interaction terms in regression. To be valid, this hypothesis must be established *before* data analysis.
Here, a significant β₃ coefficient provides evidence of HTE for subgroup S.
Leveraging ML algorithms (e.g., Causal Forests), this method discovers key subgroups directly from the data. It avoids strong assumptions and explores intricate relationships, preventing overfitting via techniques such as sample splitting for robust estimation.
Strong analytical ability demands careful execution. Exploratory analyses, in particular, need rigorous methods to prevent errors, false findings, and damage to research integrity.
Repeated subgroup testing vastly inflates the odds of a spurious, "significant" finding. This, known as p-hacking, severely undermines scientific reliability.
With 10 tests at 5% significance, the chance of a false positive is roughly 40%.
A robust Pre-Analysis Plan (PAP) is key to thwarting p-hacking. This publicly registered document, drafted prior to analysis, distinguishes planned, confirmatory tests from later, exploratory discoveries.