Laterality judgments (i.e., discerning whether a stimulus object is located on the left or right side of a reference point) take longer and are more error-prone when made from a third-person perspective (3rdPP) than from one’s own perspective (i.e., first-person perspective, 1stPP). Furthermore, 1stPP interferes with judgments made from 3rdPP, evidenced in larger performance impairment the more the to-be-adopted perspective deviates from 1stPP. It has been suggested that such interference is minimized by inhibiting the 1stPP when adopting a 3rdPP. We investigated this conjecture by having N = 40 participants switch between a task involving laterality judgments from 1stPP vs. from the perspective of a stimulus figure which could be presented in back-facing (i.e., aligned with 1stPP) or front-facing (i.e., misaligned with 1stPP by 180°) format. Laterality judgments from 1stPP were overall faster and less error-prone than from 3rdPP but involved a larger switch cost. The switch cost asymmetry occurred despite a long interval for task preparation of 1000 ms. This result is consistent with the notion of inhibition of 1stPP processing which is not resolved during task preparation. Moreover, judgment from 1stPP was substantially impaired in trials featuring a front-facing compared to a back-facing figure, a finding usually absent in single-task 1stPP laterality judgments demonstrating that frequent switching between judgments made from 1stPP and from 3rdPP makes 1stPP processing vulnerable to interference from the imagined perspective. An intervention designed to stimulate 3rdPP did not alter the results.