Placing the hands near the stimuli have been shown to modulate interference effects in conflict tasks. Liepelt and Fischer (2016) observed that the direction of this near-hand effect depends on the task demands. In a number-categorization Simon task, conflict was reduced in the proximal compared to the distal stimulus-hand condition, while the reverse pattern was true for a less complex color-discrimination Simon task. The authors explained the different directions of this stimulus-hand proximity effect by a flexible attentional weighting of bottom-up versus top-down signals depending on task demands. The present study aimed to test this hypothesis by fitting the data of Liepelt and Fischer (2016) to the Diffusion Model for Conflict (DMC). DMC assumes that stimulus-response translation in conflict tasks involves two overlapping diffusion processes, consisting of automatic distractor-based activation and controlled target-based activation. Across two experiments, model fits showed that stimulus-hand proximity selectively modulated distractor-based activation strength, while other processing components were largely unaffected by the stimulus-hand proximity. Most importantly, the direction of this modulation mirrored the opposing behavioral effects observed in the Simon tasks of the original study: stronger spatial distractor-based activation in the stimulus-hand proximal compared to the stimulus-hand distal condition in the color-discrimination task, but weaker activation in the stimulus-hand proximal compared to the stimulus-hand distal condition in the number-categorization task. These results support the idea that stimulus-hand proximity selectively modulates distractor processing depending on task demands.