In a series of experiments, we investigated how well native and non-native participants can remember what was said (item memory) and who said it (source memory). The ability to remember a particular piece of information and also from whom this information was obtained can be critical, for example, for continuing a conversation where it left off, for making an informed judgment about the information’s credibility, or even for predicting what a speaker might talk about next. However, due to attentional demands, the ability to encode such item-speaker associations may vary with language proficiency. In our experiments, native and non-native participants first heard two speakers refer to individual objects before they had to remember which objects had been mentioned (item memory) and/or who had previously named them (source memory). Visual objects had either a prototypical color (e.g., green broccoli) or were images typical for a specific speaker age (e.g., a children’s shovel). Speakers either consistently named objects of one color or objects matching their age, or they randomly named objects. While both participant groups in the color experiment associated objects with speakers successfully, a consistent pattern for item-speaker associations facilitated encoding only for native participants, while non-native participants had no such memory advantage. We will discuss in how far differences in pattern consistency and memory type can vary with language proficiency and participants’ ability to attend to aspects of the L2.