How people evaluate others’ gaze in social interactions depends not only on specific gaze directions (e.g., more positive evaluation of downward averted gaze compared to other gaze aversions) but also on the context in which the gaze patterns occur (e.g., in negative contexts, the preference for direct gaze compared to downward averted gaze is diminished). In a series of four experiments employing the ‘Video-Call Paradigm’, we wanted to further investigate the gaze-context interplay including more natural, nuanced and dynamic gaze patterns. During the task, participants watched pre-recorded video clips displaying a dyadic conversation, in which one partner was listening to a short personal narration of another person (not visible, but audible ). After each clip, participants rated the listener on relevant dimensions for social perception (e.g., empathy). The independent variable, gaze direction of the listener, was manipulated within subjects and was either static-direct or switched dynamically between direct (5 sec. interval) and downwards, sideways or upwards averted gaze (2 sec. interval). Second, emotional context was manipulated between subjects in that narrations varied comparing neutral with either sad (Exp.1), fearful (Exp.2), happy (Exp.3), or embarrassing (Exp. 4) narrations. Results suggest that listeners were differentially evaluated depending on their gaze direction and emotional context. For example, downward gaze was evaluated more positively, especially in sad contexts, while upward averted gaze was perceived as less favorable, especially in contexts of negative valence. Overall, the current studies demonstrate how different gaze patterns affect the perception and evaluation of individuals within emotional contexts.