It has been shown that phasic alerting increases congruency effects in some conflict tasks (e.g., Simon and arrow flanker tasks), but not in others (e.g., symbol flanker tasks). In attempts to find explanations for this puzzling observation, research has mainly focused on the role of spatial information processing in the alerting-congruency interaction. In contrast, the role of stimulus modality, in particular alerting signal (AS) modality, has not been considered systematically so far. In this experiment, we investigated specifically the effects of both AS modality and task stimulus modality on the alerting-congruency interaction. Our experiment had two parts: In the first part, we let participants (N = 36) perform a speeded detection task using visual and auditory stimuli of varying intensity to create intensity-matched stimuli that served as AS in the second part. In the second part, participants performed a visual or auditory Simon task, with task stimulus modality changing across blocks. Within each block, we induced phasic alerting by presenting an auditory or visual AS in half of the trials. We observed a larger congruency effect with AS than without AS. Importantly, AS modality did not modulate this interaction, suggesting that both visual and auditory AS were effective in modulating congruency effects. Interestingly though, there was some indication of an influence of task stimulus modality on the alerting-congruency interaction. This latter result may be attributed to (modality) differences in the time course of automatic response code activation, which may alter the effectiveness of phasic alerting in influencing response selection.