Age studies have shown that older individuals often have prolonged response times compared to younger individuals, whereas their accuracy rates differ only rarely. Diffusion model analyses aimed at examining the reasons underlying these findings and have revealed a consistent pattern in terms of two parameters: Older individuals have higher threshold separations (i.e., they are more conservative in their decisions) and prolonged nondecision times (i.e., they need more time for stimulus encoding or motor processes). Regarding drift rate, which is a measure of speed of information accumulation, the results of these previous diffusion model studies are, however, inconsistent. In some studies, younger individuals outperformed older individuals in terms of drift rate; in other studies, there was no difference between age groups or even a reversed pattern. In our study, we identified one moderator that seems to at least partly explain these inconsistent findings. We had 30 young participants (< 37 years) and 30 older participants (> 60 years) work on six typical response time tasks and varied task difficulty within each task. As expected, we found the typical age differences in threshold separation and nondecision time. Moreover, for drift rate, a consistent age by task difficulty interaction emerged: Even if younger individuals were superior to older individuals in terms of speed of information processing in almost all tasks, this effect was clearly smaller for the more difficult stimuli. Thus, older individuals profited from an increase in task difficulty.