Lateralized responses to non-spatial features of a stimulus are slowed-down when the stimulus is presented contralaterally as compared with ipsilaterally to the response side. Although this so-called Simon effect is found in vision and audition, a decreasing Simon effect with increasing response time levels, typically reported for the visual domain, is usually not found in the auditory modality, suggesting modality-specific mechanisms underlying spatial response conflicts. The present study investigated whether the more persistent Simon effect in the auditory modality is due to the dynamic characteristic of auditory as compared with visual stimuli, with the former typically changing over time. In an auditory (Experiment 1) and a visual (Experiment 2) accessory variant of the Simon task, a lateral distractor occurred prior to or simultaneously with a central target. Participants responded to the identity of the target via keypress and encountered dynamic (visual/auditory noise) and static (sine tone/checkerboard pattern) distractors. The analysis of averaged response times and error rates was supplemented by distribution analyses. An overall Accessory Simon Effect (ASE) was obtained in both experiments. Importantly, the time course of the ASE was comparably affected by distractor type in both modalities, decreasing less across SOAs with dynamic as compared with static distractors. Together, the findings are in line with the notion that (1) common mechanisms underly the ASE in vision and audition and that (2) the time course of the effect can be similarly affected by temporal stimulus characteristics in both modalities.