The Potential of Registered Reports for Machine Learning Modeling in Psychological Research
Tue-03
Presented by: Kristin Jankowsky
Background / Problem
There are numerous promising opportunities for the use of machine learning (ML) algorithms in psychological research. ML algorithms have shown to be especially advantageous for maximizing predictive accuracy, combining a large number of heterogenous predictor variables or tackling a high predictors/participant ratio, thereby lessen a model’s potential overfit to the data. However, across a wide range of disciplines, critiques are currently accumulating, demonstrating that ML models are often implemented incorrectly and interpreted overoptimistically. One problems stand out in particular: Incorrect model validation usually leads to bias in favor of more complex and flexible algorithms as those are better equipped to recognize specific data patterns as well as exploiting any spillover of information between training and testing data. This could lead to the dissemination of false discoveries or to the development of unsubstantiated theories. Consequently, these overoptimistic or biased ML models do not live up to their expectations if correctly validated.
Approach / Solution:
In this talk, I propose a more open debate and culture of mutual scrutiny to enhance transparency and avoid common pitfalls in ML. One possible way to achieve this is by making methodological feedback prior to conducting the analyses the norm. Whereas there are initiatives that actively invite and encourage methodological criticism (e.g., the Red Team Market), an even more obvious solution would be to employ registered reports. At first glance, this proposal seems to counteract the empirically driven and flexible nature of ML algorithms. Fittingly, ML modeling and registered reports rarely have been combined in psychological research so far. However, I argue that many aspects concerning data cleaning, variable transformation, handling of missing data, etc. can be registered in ML studies the same way as in every other study. Also, the settings for data-driven hyperparameter tuning can also be defined in advance.
Discussion / Implications
Based on the experience with a registered report including ML modeling within clinical psychology, I aim to highlight the compatibility of ML modeling and open science practices. In the ideal case, registered reports could be a remedy for many pitfalls in ML-based research because researchers a) get high quality feedback prior to analyses, b) have to think about hyperparameter ranges and modeling decisions instead of trying out excessively, c) do not need to produce flashy results or necessarily show the superiority of ML models and d) have to ensure the transparency and reproducibility of modeling decisions. In contrast to this very positive view, I also discuss current limitations of registered reports. For example, albeit that the number of journals offering registered reports is constantly growing, there are still research fields for which they are less common, especially when reanalyzing existing data. Additionally, the more complex the algorithms and data get, the more complicated it may be to truly register all analyses beforehand which in turn requires flexible solutions or hybrid models. I will conclude my presentation with recommendations for combining ML-based modeling and open science practices in general.
There are numerous promising opportunities for the use of machine learning (ML) algorithms in psychological research. ML algorithms have shown to be especially advantageous for maximizing predictive accuracy, combining a large number of heterogenous predictor variables or tackling a high predictors/participant ratio, thereby lessen a model’s potential overfit to the data. However, across a wide range of disciplines, critiques are currently accumulating, demonstrating that ML models are often implemented incorrectly and interpreted overoptimistically. One problems stand out in particular: Incorrect model validation usually leads to bias in favor of more complex and flexible algorithms as those are better equipped to recognize specific data patterns as well as exploiting any spillover of information between training and testing data. This could lead to the dissemination of false discoveries or to the development of unsubstantiated theories. Consequently, these overoptimistic or biased ML models do not live up to their expectations if correctly validated.
Approach / Solution:
In this talk, I propose a more open debate and culture of mutual scrutiny to enhance transparency and avoid common pitfalls in ML. One possible way to achieve this is by making methodological feedback prior to conducting the analyses the norm. Whereas there are initiatives that actively invite and encourage methodological criticism (e.g., the Red Team Market), an even more obvious solution would be to employ registered reports. At first glance, this proposal seems to counteract the empirically driven and flexible nature of ML algorithms. Fittingly, ML modeling and registered reports rarely have been combined in psychological research so far. However, I argue that many aspects concerning data cleaning, variable transformation, handling of missing data, etc. can be registered in ML studies the same way as in every other study. Also, the settings for data-driven hyperparameter tuning can also be defined in advance.
Discussion / Implications
Based on the experience with a registered report including ML modeling within clinical psychology, I aim to highlight the compatibility of ML modeling and open science practices. In the ideal case, registered reports could be a remedy for many pitfalls in ML-based research because researchers a) get high quality feedback prior to analyses, b) have to think about hyperparameter ranges and modeling decisions instead of trying out excessively, c) do not need to produce flashy results or necessarily show the superiority of ML models and d) have to ensure the transparency and reproducibility of modeling decisions. In contrast to this very positive view, I also discuss current limitations of registered reports. For example, albeit that the number of journals offering registered reports is constantly growing, there are still research fields for which they are less common, especially when reanalyzing existing data. Additionally, the more complex the algorithms and data get, the more complicated it may be to truly register all analyses beforehand which in turn requires flexible solutions or hybrid models. I will conclude my presentation with recommendations for combining ML-based modeling and open science practices in general.