Task-irrelevant sound is known to disrupt short-term memory either by interfering with specific processes or through a diversion of attention. This study investigates the disruptive effect of whispered speech to test diverging predictions of these accounts. According to an interference-by-process account, whispered speech should be less disruptive, because the reduced temporal fine structure and lower amplitude modulations may provide weaker order cues to the auditory system, thus inducing less interference with seriation processes. In contrast, the attentional account predicts more disruption by whispered speech, due to the enhanced listening effort to process semantic information of whispered utterances (when presented in a comprehensible language), and because the contents of whispered speech may be considered potentially more important. In Experiment 1, participants were asked to remember visually presented letters while either loud or whispered speech was presented. The speech recordings consisted of either a single German word that was played repeatedly (steady-state) or full German sentences (changing-state). Recall accuracy was lower in the presence of changing-state speech, indicating interference-by-process. However, regardless of the state, whispered speech was found to be more disruptive than loud speech, thus indicating an additive effect of attentional capture. To test the listening-effort account, the same speech materials were presented to English participants who did not understand German (Experiment 2). While the changing-state effect was equivalent to Experiment 1, whispered speech was less disruptive than loud speech to the English sample, indicating that psychoacoustic properties may dominate when no additional listening effort is demanded by a foreign language.