15:30 - 17:00
Location: Senate Room (19/F LAU)
Chair/s:
Le Bao
Discussant/s:
David Broska
Na Liu - Information conditions govern the validity of AI-generated experimental data across inferential targets
Plamen Akaliyski - When AI Thinks about Culture: Comparing Language-Based and Empirical Measures of Individualism-Collectivism
Leo Yang - Scaling Reproducibility: An AI-Assisted Workflow for Large-Scale Replication and Reanalysis
Simon Maier - Replication and Specification Range Analysis with AI Agents
Submission 73
Scaling Reproducibility: An AI-Assisted Workflow for Large-Scale Replication and Reanalysis
Panel 1-Senate Room (19/F LAU)-03
Presented by: Leo Yang
Yiqing Xu 2, Leo Yang 1
1 Hong Kong Baptist University
2 Stanford University
Computational reproducibility is central to scientific credibility, yet verifying published results at scale remains costly. We develop an AI-assisted workflow for automated full-paper replication -- retrieving materials, reconstructing environments, executing code, and matching outputs to point estimates reported in regression tables. We define a universe of all empirical and quantitative papers from the three top political science journals (2010--2025) and measure stated data availability using automated extraction. For a stratified sample of 384 studies, we apply the workflow to conduct full-paper replication, totaling 3,523 empirical models. We find that journal verification requirements, combined with data archiving mandates, drive reproducibility: the share of fully or largely reproducible papers rises from 20.8% before DA-RT adoption to 82.5% after, and conditional on accessible replication packages, 92.1% of papers are fully or largely reproducible (234/254). As a secondary application, we apply standardized IV diagnostics to 84 studies (597 IV specifications among 1,910 replicated models), illustrating how automated execution enables systematic reanalysis across heterogeneous empirical settings.