Submission 114
The Trade-off Between Precision and Robustness in Explainable AI: Do People Appreciate It when an AI Uses Generalised Concepts?
SymposiumTalk-04
Presented by: Romy Müller
What does an AI model actually see in an image it classifies? Concept-based explainable AI (C-XAI) can reveal this by presenting image snippets from the dataset that, according to the AI, resemble the classified image. While C-XAI evaluations typically aim for these snippets to be as coherent as possible, AI systems should also be robust. Thus, some generalisation is desirable. However, do people appreciate this, given the inevitable trade-off between coherence and generalisation? If so, generalisation should lead to high ratings of AI performance, given that the meaning of the image is retained. In the present study, participants rated an AI that classified images of railway trespassers as dangerous or not. The images varied in a relevant feature (a person’s relation to the tracks) and a less relevant feature (i.e., a person’s specific action). Participants saw C-XAI explanations consisting of snippets in different conditions: the snippets precisely matched the original image (i.e., same action, same relation to tracks), generalised over the relevant feature (i.e., randomised relations to tracks), generalised over the less relevant feature (i.e., randomised actions), or systematically got the less relevant feature wrong (i.e., consistent but different action). Participants rated the AI highest when snippets closely matched the original image. While they strongly dispreferred generalisations over the relevant feature, they rated generalisations over the less relevant feature lower than precise matches, and as low as systematic mistakes. These findings suggest that people may not always favour robustness, but instead expect AI systems to be maximally precise.