The ability to filter task-irrelevant distractors that are conflicting with a current task is a fundamental function of selective attention. One of the most prominent tasks for investigating response conflicts in the visual domain is the Eriksen flanker task in which a central target is paired with a distractor that is either associated with a conflicting response (incompatible) or a compatible response. The response conflicts are then measured as differences in response times and error rates between incompatible and compatible trials. Rather than comparing only mean differences, diffusion models account for the entire process of response conflict. While there is evidence that auditory flanker effects are based on a similar conflict mechanism as in the visual modality, it is unclear to what extend diffusion models are able to account for the conflict process in the auditory domain. The present study investigates this by fitting the diffusion model of conflict task (DMC) and the shrinking spotlight model (SSP) to visual and auditory data using the dRiftDM package for R. Both models revealed longer encoding processes, prolonged but weaker response conflicts and slower evidence accumulation for the auditory modality while the amount of evidence required to make a decision did not differ between modalities. The DMC was found to generally overestimate auditory conflict effects, while the SSP overestimated the conflict only for the fastest responses. Consequently, the SSP provided the better fit for auditory conflict effects.