Capacity limitations in visual tasks can be observed when the number of task-related objects increases. An influential idea is that such capacity limitations are determined by competition at the neural level: two objects that are encoded by shared neural populations interfere more in behavior than two objects encoded by separate neural populations. However, the neural representational similarity of objects varies across brain regions and across time, raising the question of where and when competition determines task performance. Furthermore, it is unclear whether the association between neural representational similarity and task performance is common or unique across tasks. Here, we used fMRI, MEG, and deep neural networks (DNN) to provide a detailed spatiotemporal association between neural representational similarity and performance on two tasks, both involving the same set of two-object displays. In the visual search task, participants located a pre-cued target object, while in the same/different task participants indicated whether the two objects were the same or different. Separate groups of participants viewed the individual objects in neuroimaging experiments to establish the neural representational similarity of the object set. For both tasks, the pairwise behavioral interference was correlated with neural representational similarity throughout the visual system, from 100 ms after onset, and in all layers of a DNN. Semi-partial correlation analysis, however, revealed task-specific associations, with the same/different task uniquely associated with early/posterior neural similarity, and the visual search task uniquely associated with late/anterior neural similarity. These results provide a detailed task-specific mapping between neural similarity and behavioral interference.