Asymmetrical switch costs (i.e., larger costs of switching from a “weaker” to a “stronger” task than vice versa) have been attributed to inhibition of a dominant competitor set (i.e., cognitive control account), task-specific consumption of cognitive resources (i.e., sequential difficulty account), or differential susceptibility to (long-term memory-based) interference in task repetition trials due to task-set competition (i.e., associative interference account). We investigated the source of asymmetrical switch costs of when switching between laterality judgments made from a third-person perspective (3PP) vs. from a first-person perspective (1PP) by means of a probe task method. Specifically, we intermixed trials of a Simon task to examine inhibition of the “direct route” (as a process associated with the inhibition of the dominant 1PP) – assumed to activate response codes that spatially correspond with stimulus laterality according to the 1PP – and to compare resource consumption on 1PP and 3PP trials. Moreover, intermixing trials of the Simon (probe) task allowed us to analyze interruption costs (i.e., 1PP and 3PP task performance following a Simon task trial). We observed asymmetrical switch costs with larger costs when directly switching from the 3PP to the 1PP task. Simon task performance provided no clear evidence for inhibition of the direct route or for larger resource depletion after 3PP trials. Comparing 1PP and 3PP performance after Simon task trials revealed asymmetrical interruption costs comparable to the switch cost asymmetry, however, a result uniquely predicted by the associative interference account.