11:20 - 13:00
P2-S41
Room: 0A.07
Chair/s:
Piotr Koc
Discussant/s:
David S Siroky
To Harmonize or Not? Research Design for Cross-Context Learning
P2-S41-3
Presented by: Cyrus Samii
Cyrus SamiiAnna Wilke
New York University
To enhance generalizability, researchers increasingly study the same question in multiple contexts. Efforts at cross-context knowledge accumulation range from meta-analyses of studies that differ widely in design to harmonized initiatives that implement nearly identical experiments. Recent formal analyses have emphasized the value of harmonization, although this runs against intuitions that lead researchers to introduce heterogeneity intentionally for robustness. We use a decision-theoretic framework to resolve this apparent contradiction and understand the conditions under which cross-study harmonization versus diversification of research designs improves learning. We consider learning about a mean effect from a global distribution and learning about how effects differ across locations. Under a minimax regret objective, we show that diversification is useful because it maximizes precision for learning about a mean effect and manages ambiguity about potential biases. The benefit of harmonization comes from the ability to make more precise comparisons of effects in situations where bias concerns are secondary. The analysis has implications for other design questions, such as the choice of outcome measures. Our minimax regret analysis follows recent work in econometrics and offers an introduction for political scientists.
Keywords: Research design, causal inference, decision theory, minimax regret, experiments

Sponsors