Perceptual load theory proposes that processing of distractor stimuli is determined by the level of perceptual load, with low distraction (assessed by flanker interference) at higher load levels. However, He and Chen (2010) observed that familiar faces caused flanker effects even at higher load levels compared to unfamiliar ones. We expected a comparable pattern for emotional faces, that is, happy or angry faces should distract still at load levels at which neutral faces ceased to distract. In our experiments, we asked participants to categorize names according to gender. The target was presented with 1, 3, 5, or 7 pseudo-names (i.e., the variation of perceptual load) accompanied by a face distractor. In Experiment 1a, we manipulated emotion: happy vs. neutral; in Experiment 1b, we compared angry vs. neutral. Both happy and angry faces caused significant flanker effects at load levels 1 and 2, which were no longer significant at levels 3 and 4. Unexpectedly, neutral faces’ congruency effects changed depending on whether they were presented with happy or angry faces, thereby seemingly indicating that neutral faces have less distraction potential than happy faces but more distraction potential than angry faces. To clarify, Experiment 2 was a full replication of Experiment 1a,b. Again, happy and angry faces showed comparable results: we found clear flanker effects at level 1 and 2, reduced (but still significant) effects at the higher levels. The neutral faces, however, were again inhomogeneous. We will discuss these results considering how neutral faces might be ambiguously perceived depending on context.