A person’s gaze behavior has considerable influence on how they are evaluated by others. However, interpreting someone else’s gaze can also depend heavily on the context. Our aim was to systematically and comprehensively explore the "semantics of gaze" in different contexts using a two-step procedure (qualitative-quantitative). Participants watched short video clips of one person listening to an autobiographical episode narrated by another person. We extensively manipulated the listener's gaze behavior in terms of gaze direction (direct, down, up, side), frequency of changes in gaze direction and blink frequency (every 2, 4, 6 seconds). In addition, emotional context was manipulated by the valence of the narrated episodes (neutral versus negative). In a first, exploratory study, participants freely described which characteristics, feelings, and intentions they attributed to the listener in each condition. This allowed us to identify several relevant aspects of person perception (e.g., attentiveness, likability, calmness) and construct respective rating scales. In a second, quantitative study, participants used these scales to indicate their impressions after each of the videos. Results suggest that listeners were systematically and differentially evaluated depending on their gaze behavior. For example, rapid blinking and rapid changes in gaze direction were rated more negatively than slower gaze behavior in various respects (e.g., as inattentive). Downward gaze was evaluated more positively (e.g., as empathic) than other gaze aversions, especially in the emotionally negative context. Overall, our studies contribute to deciphering the meaning of gaze behavior in different emotional contexts.