We tested the hypothesis that prospective and retrospective metacognitive judgments of performance in memory tasks accompanied by auditory distraction reflect general beliefs about the harmfulness of noise rather than task-specific distraction effects. Previous studies have shown that people can correctly rank different types of sounds from steady to changing as more or less disruptive, but it remains unclear whether these assessments adapt to the nature of the memory task. In a series of experiments, participants performed a missing-item task, which is thought to be immune to some of the distraction effects, and a probe serial-recall task, which is more sensitive to the distraction effects. In the probe serial-recall task, we also manipulated task difficulty by varying the memory set size, allowing us to test whether metacognitive judgments of distraction are modulated by cognitive load. Before and after task, participants predicted and postdicted the extent to which different types of background sounds impair their performance. The results suggest that people’s judgments of distraction depend on whether the sound is changing or steady, that these patterns are largely the same regardless of the type of the memory task, and that some metacognitive patterns are modulated by cognitive load. We suggest that metacognitive monitoring of distraction is sensitive to both the characteristics of the sound and the memory task it accompanies, but not to the interactive patterns of the two.