The irrelevant speech effect (ISE) refers to the disruption of verbal short-term memory by background speech. It has been studied in numerous laboratory experiments, which typically, exhibit low ecological validity compared with a real-life environment with multiple speech sources. Therefore, we set out to investigate the ISE in a virtual reality (VR) scenario permitting to position both target and interfering speakers in audio-visual space. In Experiment 1, participants were exposed to either a slow (1 syllable / 700-ms; N = 39) or a fast (1 syllable / 200-ms; N = 40) presentation rate of a sequence of task-irrelevant letters seamlessly uttered by either one or three distractor speakers distributed in space while they had to memorize a sequence of digits pronounced (at a rate of 1/s) by a target speaker. The results indicate that while the basic ISE is robustly replicated in VR, more complex spatial effects based on the auditory segregation of speakers failed to reach statistical significance. In Experiment 2, the auditory presentation of the to-be-remembered material (target speaker straight ahead) was contrasted with its visual presentation (digits presented on a virtual laptop) while improving the spatial layout of the audio-visual scene (N = 60). The results are discussed in terms of the role of spatial auditory streaming in modulating the memory impairment produced by distracting speech.