Since the outbreak of the COVID-19 pandemic the use of face masks in order to contain the risk of infection has become common. However, face masks also affect social interactions, for example by impeding face recognition. In an online study based on a within-subjects design, 36 participants observed masked and unmasked faces before they performed a face recognition task with masked and unmasked faces. In contrast to prior studies, data were analyzed using the diffusion model (Ratcliff, 1978) which allows to differentiate between cognitive processes underlying face recognition performance. We hypothesized that the poorer recognition performance for masked faces and the incongruency effect (between study and test phase) are caused by differences in drift rate, the diffusion model parameter which reflects speed of information processing. The results confirmed the hypotheses. More specifically, drift rate was highest when faces were unmasked at both study and test phase and lowest in the incongruent mask conditions. These findings can be explained by the impairment of holistic representation of masked faces and the additional violation of the encoding specificity principle for incongruent mask conditions. Our study shows that the use of face masks impedes face recognition via a reduced drift rate.