Human and non-human primates interact with their environment by manipulating objects. This requires planning and executing reaches to transport the hand to object-relative positions on the objects. Previous studies showed that frontal and parietal lobe areas encode reach goals in coordinate systems anchored to different parts of the subjects’ body, such as hand position or gaze direction. Such egocentric encoding is a well-known property of brain areas along the dorsal pathway, whereas allocentric encoding with a reference frame centered on an object, is not. Different to previous experiments, which typically only demanded egocentric spatial processing during movement preparation, we designed a task where monkeys memorized an on-the-object target position and then planned a reach to this position after the object re-occurred at variable location with potentially different size. We found allocentric (in addition to egocentric) encoding in the dorsal stream reach planning areas, parietal reach region and dorsal premotor cortex, which is invariant with respect to the position, and, remarkably, also the size of the object. This means that neurons in the same areas can encode object-centered allocentric spatial information, independent of object location and object size, or egocentric information, depending on dynamically changing cognitive demands. This dynamic adjustment suggests that the prevailing frame of reference is less a question of brain area or processing stream, but more of the cognitive demands. Such dynamic shift seems inconsistent with a functional segregation for ego- and allocentric encoding between processing streams.