In this project we test an evaluative conditioning effect (EC; a change in liking due to paired presentations) across a series of five replication studies. These studies represent different realizations of online- and lab-settings and include: (1) a lab experiment with an experimenter present (university sample), (2) a lab experiment without an experimenter (university sample), (3) an online experiment on computers (university sample), (4) an online experiment on a computer or laptop (provider sample), (5) an online experiment on a smartphone (provider sample). Adjacent studies are identical in all aspects (experimental program, incentives, language etc.) but one that is systematically varied. Statistical approaches are used to identify and deal with unintended, but unavoidable, difference such as biased drop-out or non-compliance.

This research is part of a larger project in which we aim to systematize possible causes for (non-) replication and develop design and analysis techniques to identify them within specific fields based on the Causal Replication Framework (CRF, Steiner, Wong, & Anglin, 2019) which systematizes study characteristics. Differences in results across replications (and other cross-study comparisons) are typically difficult to interpret because several factors vary at once. Even in EC, a relatively well-understood effect, unintended differences (for example in attention or personality traits) are theoretically plausible to affect the outcome.

We will report results from the five replication studies with a focus on how closely the EC effect is replicated. We highlight challenges in avoiding unintended study differences and offer strategies to deal with them.