Despite the importance of reliable results, replication rates have been unexpectedly low. For almost a decade, methodologists have been working to explain these low rates and understand the factors that contribute to reliable results. One notable study conducted by Altmejd et al. (2019) used a machine-learning model to predict replication success. They considered statistical indicators, such as p-values, as well as non-theoretical indicators like the number of authors. We expand upon their findings by incorporating theory-driven predictors from the four validities-framework (Fabrigar et al., 2020) that closely relate to the design of experimental research. For internal validity, we consider, for example, attrition rates. For external validity, we include, for example, differences in sample characteristics. For construct validity, we include, for example, the extent of validation for the measures used. Lastly, for statistical conclusion validity we include experimental design factors such as the use of within- or between subjects manipulations. Our goal is to provide a more comprehensive understanding of the factors that are associated with replication success, which may ultimately enhance the reliability of empirical research.
Altmejd, A., Dreber, A., Forsell, E., Huber, J., Imai, T., Johannesson, M., Kirchler, M., Nave, G., & Camerer, C. (2019). Predicting the replicability of social science lab experiments. PLoS ONE, 14(12), 1–18. https://doi.org/10.1371/journal.pone.0225826
Fabrigar, L. R., Wegener, D. T., & Petty, R. E. (2020). A validity-based framework for understanding replication in psychology. Personality and Social Psychology Review, 24(4), 316–344. https://doi.org/10.1177/1088868320931366