Visual perspective taking (VPT) provides people with insights into how objects look from another’s point of view. Here, I will review evidence for a recent series of studies that VPT can be understood as a (quasi-)perceptual phenomenon, in which another’s perspective “stands in” for own sensory input and drives perceptual decision making. Using a variant of the mental rotation task, these studies show that participants can recognize items oriented away from themselves more rapidly when these items appear in a more canonical orientation to an incidentally presented another person (and slower when oriented even further away from them). These effects are of large effect size and observed even when the other person is completely passive and task-irrelevant. They therefore show that people spontaneously derive the content of another’s perspective in a form that can drive perceptual processing like one’s sensory input. They are affected by several factors, such as whether participants intentionally take the other’s perspective or do so spontaneously, whether the other person attends to the same object as oneself, or whether one is in the presence of another human or an object to which various levels of mental states can be ascribed (e.g., robots, inanimate objects). Together, these findings argue for a framework in which perceptual anticipations of another’s perspective underlie social cognition, helping us to understand not only how other’s view the world, but also letting us vicariously explore how they would respond to it.