In recent years, the Simon effect has been intensively studied using multiple stimulus locations along the horizontal or vertical axis of the screen’s center. We recently implemented a version of the Simon task in which a single stimulus setup was compared to a multiple stimulus setup containing a perceptual set of identical stick-figure manikin stimuli. With this paradigm, we typically observe two kinds of Simon effects, i.e. one along a global reference frame (based on the screen’s side) and local reference frame (based on the side of manikin stimulus). In the present set of experiments, we investigated the role of task difficulty on the emergence of both Simon effects when a perceptual set of identical manikin stimuli was used. In Experiment 1, the influence of the perceptual distribution and clarity within the set of identical manikins was investigated. Experiment 2 addressed the difficulty of the task-relevant stimulus feature in the Simon task (i.e. color vs. shape classification). Taken together, the results show that both kinds of Simon effects are differently modulated by the different experimental manipulations of difficulty.