When we listen to others’ stories, our nonverbal behavior can vary depending on the emotional context of what we hear. But are these differences distinct enough for an observer to infer the emotional context solely from the listener’s behavior? And if so, which aspects of gaze behavior do people rely on to make such judgments? To address this question, we filmed participants while they listened to (supposedly) autobiographical narrations told by a confederate. The narrations described experiences of different emotional valence (neutral, negative, positive, and embarrassing). In a subsequent online study (n = 82), we then presented 30-second clips of the previously filmed listeners, without any information about the story content, and asked naïve participants to guess the emotional content the observed others were listening to. Participants identified the correct emotional context at rates above chance, though overall accuracy was modest. Accuracy differed significantly across emotional contexts and also showed substantial variability between the listeners. In follow-up studies, we explored which features of listeners’ nonverbal behavior contributed to accurate judgments. Together, these results suggest that listeners’ nonverbal behavior carries meaningful information about the emotional dynamics of an interaction, even when detached from its verbal content.