It is well established that P3 latencies increase with age as part of cognitive aging. Investigating these age-related latency differences requires numerous methodological decisions, resulting in pipelines of great variation ultimately limiting the comparability of results from different studies. The aim of the present work is to investigate the effects of different analytical pipelines on the age effect in real data. Therefore, we conducted a multiverse study and varied the low-pass filter setting (4 Hz, 8 Hz, 16 Hz, 32 Hz), the latency type (area vs. peak), the level of ERP analysis (single participant vs. jackknifing) and the extraction method (manual vs. automated). 30 young (18 – 21 years) and 30 old (50 – 60 years) participants completed three cognitive tasks (Nback task, Switching task, Flanker task), while an EEG was recorded.
The results show that different analysis strategies can have a tremendous impact on the detection and magnitude of the age effect, with effect sizes ranging from 0% to 96% explained variance. Likewise, regarding the psychometric properties of P3 latencies, we found that the reliabilities fluctuated between rtt = .13 and 1.00, while the homogeneities ranged from rh = -.18 to .86. Based on predefined criteria, we recommend applying a tight low-pass filter and performing manual extraction, especially when dealing with noisy data and using peak latencies at the individual participant level. Furthermore, our findings add to the evidence that jackknifing combined with peak latencies can lead to non-informative results, as illustrated here by severe overestimation of effect sizes.