During conversations, we often interpret our interaction partner’s gaze to gauge their level of interest, understanding and empathy towards our utterances. The present study investigated which gaze behaviors are evaluated favorably in a listener – depending on the emotional character of the conversation – and whether listeners actually display this ‘optimal’ gaze behavior in naturalistic conversation settings. To gain a comprehensive understanding, we included different emotional contexts (e.g., sad, happy) and different aspects of gaze behavior such as direct gaze, frequency of gaze aversion, etc. We conducted several experiments, first maintaining high experimental control by presenting pre-recorded videos of individuals listening to brief autobiographical narrations. The participants evaluated the listener’s gaze behavior on different dimensions (e.g., empathy). We then moved to dyadic face-to-face interactions in a laboratory setting, where we recorded participants' own gaze behavior during conversations with a confederate. Our findings underscore the critical influence of contextual factors on what constitutes appropriate gaze behavior in a listener and provide valuable insights into the dynamics of gaze behavior in everyday social interactions.