Real-world events of increasing societal polarization have attracted the attention of political scientists and pundits over the last decade. While there is a vast body of scholarly knowledge on the extent of societal polarization and its consequences, such as partisan prejudice and the embrace of undemocratic behavior, our interpretations of this phenomenon is derived from almost exclusively a single survey item: the like-dislike scale of party supporters. Despite the potential implications of societal polarization for democracies, the existing measurement tools have not been sufficiently studied. In this research, we argue that the measurement tools used to estimate societal polarization are crucial for understanding out-partisan hostility. We propose new measurements to capture out-partisan hostility and compare them with the traditional measurement using like-dislike scale. We test and validate these new measures via an experimental study, where respondents are randomly asked different measurement items to capture affect towards party supporters. Subsequently, we experimentally investigate how different measurement tools used to capture societal polarization influence our interpretations of whether societal polarization leads to undemocratic behavior and discrimination against out-partisans. Our findings have important implications for how we study societal polarization, its nature and consequences.