This study focuses on the relative contributions of attentional and sensory deficits in determining age-related impairments in selective listening. Young and older adults were tested in two consecutive sessions through a task-switching variant of dichotic listening and psychoacoustic methods, including pure tone audiometry and minimum audible angle. These methods were used to assess hearing thresholds in the frequency and the spatial domains. During each session, we determined participants' ability to intentionally switch auditory selective attention in space. In each trial, two different number words were simultaneously presented from two sound sources. Guided by visual cues, participants categorized the magnitude of the target stimulus at the to-be-attended location, while ignoring the distractor stimulus at the competing location. The results showed comparable reaction times and error rates between the two age groups, with similar practice-related benefits across sessions. In both groups, responses were slower and more error-prone with the target stimulus location switching between sequential trials (i.e., switch cost) and when the target-distractor pair elicited opposite responses compared to congruent responses (i.e., congruency effect). However, the congruency effect in error rates was greater in older adults than in young adults. The observed performance differences may be due to a reduced ability to selectively allocate auditory attention in space. Here, we discuss how age-related sensory deficits may account for at least part of the behavioral effects in this selective listening task. Addressing both attentional and sensory factors may contribute to more effective and targeted interventions to mitigate age-related impairments in auditory selective processing.