This study examines the effectiveness of human oversight in counteracting discriminatory outcomes in AI-aided decision making for sensitive tasks. A mixed-method approach was employed, pairing a quantitative behavioural experiment with qualitative analyses through interviews and workshops. The experiment involved 1400 professionals in Italy and Germany, making hiring or lending decisions with either fair or biased AI-generated recommendations. Results indicate that human overseers followed AI advice in 55% of cases, irrespective of AI bias. Decision makers were prone to endorse AI suggestions that aligned with their discriminatory preferences, especially favouring candidates similar to themselves. Qualitative insights reveal the background to those decisions in terms of bias awareness, fairness perception, and ethical norms. In conclusion, human oversight alone is inadequate to prevent AI-induced discrimination. A comprehensive approach, integrating fair AI programming, norms to guide oversight, and decision maker diversity, is essential for unbiased AI-enhanced decision making processes.