We constantly interact with our environment. We have therefore organized objects in these environments such that they serve our daily activities. For example, the toothbrush we usually store close to the toothpaste and both are located above the sink where we brush our teeth. The sink therefore “anchors” predictions about the identity and location of smaller objects that we tend to interact with forming meaningful and functionally relevant sub-units of scenes, so-called "phrases". We believe that this hierarchical structure of objects in scenes is action driven and strongly engrained into the mental representations of our world. In this talk, we will highlight recent work from the lab that provides further evidence for the strength of actions in structuring both our environment and our mental object representations. For instance, we will show that so-called anchor objects – which are usually larger, and stationary – guide both attention and actions in 3D virtual environments, particularly when the objects we want to interact with are not visible and require opening cabinets or drawers. They are also key components of a scene when judging its action affordance up to the point that their presence in a scene affects lexical decision judgments of action words. Recording EEG while participants were presented with isolated objects further revealed that the representational space of objects decoded from brain activity closely follows the hierarchical structure we propose for visual scenes. It therefore becomes increasingly clear that actions structure not only objects in scenes but also in our minds.