Understanding nonverbal cues is pivotal for social interaction, enabling the inference of other’s mental states. Studies that investigated these nonverbal cues showed that reflexive gaze-guided attentional orienting can be modulated by contextual factors like action expectations (Perez-Osorio et al., 2015, 2017); however, the cognitive mechanisms underlying the processing of incongruent gaze shifts remain unclear. We propose that cognitive control modulates the impact of irrelevant gaze shifts during task execution. In this context, we measured the effects of task-related expectations on the processing of the agent’s signals in performance (RTs, error rates, and eye movements) and EEG correlates (N2 component and theta activity) of cognitive conflict. We expected (1) top-down modulation of the attentional orienting according to task goals, (2) higher performance cost to incongruent social signals reflected in cognitive conflict correlates. Results showed that participants followed the agent’s gaze, revealed by faster response times to targets congruent with head/gaze direction relative to incongruent locations (in all Experiments). Additionally, incongruent (relative to congruent) head/gaze shifts elicited higher error rates (Experiments 2-4), larger curvatures in eye-tracking trajectories (Experiment 3), larger N2 amplitudes and higher Event-Related Spectral Perturbation (ERSP) amplitudes (Experiment 4). Participants followed the agent’s gaze/head shifts, but goal-oriented expectations modulated attentional orienting to task-relevant locations independent of gaze direction. This shows that cognitive control modulates the impact of irrelevant social signals, as reflected by performance and EEG cognitive conflict markers. Our findings suggest that humans process irrelevant signals from artificial agents, which impacts complex collaborative settings, either improving or hindering performance.