Examining participant diversity in psychological research
Tue—HZ_10—Talks4—3801
Presented by: Anna Thoma
Who are the people who have been studied in psychological research for more than 120 years? Previous attempts at characterizing the demographic diversity of participants were limited to selected factors (e.g., region or gender), research subfields, or specific timespans. Despite its relevance for the generalizability of empirical findings, psychological research is yet missing an encompassing account of participant diversity and its evolution. Using natural language processing, we analyze more than 230.000 full texts of articles published in psychological journals between 1894 and 2022. Specifically, we use state-of-the-art large language models to investigate the reported composition of participant samples considering age, gender, ethnicity, language, region, education, and socioeconomic status. To evaluate our methodological approach, we compare the extracted information between the language model and human coders in a subset of articles. Our findings help to develop a better understanding of the empirical history of psychological research, highlight the relationship between diversity and scientific reporting conventions, and provide new insights into the validity of using large language models to extract content from scientific articles.
Keywords: diversity, LLM, psychological research