When Do Factorial Survey Experiments Predict Real Behavior?

A Norm–Cost Framework of Predictive Validity

Factorial survey experiments (FSEs) are widely used to study behavioral intentions and are often assumed to exhibit high predictive validity with respect to real-world behavior. However, validation studies comparing vignette responses with observed actions report highly heterogeneous results. This paper develops a norm–cost framework to explain when and why FSEs successfully predict actual behavior.

We review 16 validation studies across domains including hiring decisions, deviant behavior, mobility choices, corporate compliance, and digital privacy. Two dimensions of predictive validity are distinguished: absolute prediction of behavioral levels and relative prediction of causal determinants. Each study is classified according to the dominant normative pressure and behavioral costs involved.

The findings reveal systematic patterns. In low-cost or weakly normative contexts, FSEs frequently identify relevant causal drivers despite overstating behavioral levels. In contrast, in high-norm, high-cost situations—such as hiring discrimination and regulatory compliance—vignette responses strongly diverge from observed behavior, misrepresenting both behavioral rates and underlying determinants.

These results suggest that the behavioral external validity of factorial survey experiments is conditional rather than general. Their predictive utility depends critically on the interplay between normative expectations and real-world costs. The norm–cost framework provides guidance for the design and interpretation of future experimental studies.