17:00 - 18:30
Location: G08
Chair/s:
Daniel Flynn
Submission 42
Validating self-reported, human coded, and automated measures of emotion: An application to populism and fake news in Europe
PS6-G08-03
Presented by: Daniel Flynn
Daniel FlynnNina Wiesehomeier
IE University
Emotion features prominently in leading theories of public opinion, comparative politics, and international relations. Developing valid measures of emotions is therefore a pressing challenge for political scientists. We examine the validity of three leading approaches: self reports, human coding of open-ended survey responses, and automated analysis of open-ended responses. We do so using data from two randomized experiments in Spain and sentiment scores generated by the large language model ChatGPT. We present three sets of analyses. First, we explore the association between emotion scores generated by all three approaches. Second, we assess construct validity by estimating the effect of a randomized blame attribution treatment --- taken from the populism literature and designed to provoke strong emotional responses --- on emotions as measured by all three approaches. Finally, we assess predictive validity of each approach by examining the downstream effects of emotions on belief in fake news. These analyses offer novel methodological and substantive insights. Methodologically, we highlight best practices for measuring emotions and, more generally, evaluate the validity of ChatGPT for text analysis. Substantively, we offer evidence into the role of emotion in the psychology of fake news.