The Simon task shows is used to investigate how task-irrelevant spatial features influence performance. In the present study, we combined both, findings of agency manipulation as well as spatial layout in an enriched Simon task. Our experiments aim at investigating whether implied agency information modulates the Simon effect and if so comparable for horizontal and vertical layout. Using drawings of stick-figure manikins holding a colored ball in its left or right hand displayed at the screen’s center. Agency labels (“me”, ”friend” or ”colleague”) were presented in the stick-figure manikin’s center. Both horizontal and vertical versions of the Simon task were used. It was hypothesized that perceived agency modulates the horizontal and vertical Simon effect differently. Results of Experiment 1 confirmed this prediction, influences of labels were only seen in the horizontal version of the Simon task. To replicate and pin down the time course, Experiment 2 used a stimulus onset asynchrony.