From Pixels to Words: Leveraging Explainability in Face Recognition through Interactive Natural Language Processing

TL;DR

This paper introduces an interactive explainability framework for face recognition systems that combines XAI and NLP techniques, providing natural language and visual explanations without compromising recognition accuracy.

Contribution

It presents a novel interactive framework that enhances face recognition interpretability using natural language explanations and visual heatmaps, maintaining high accuracy.

Findings

Effective natural language explanations generated

User interaction improves understanding of FR decisions

No decrease in face recognition performance

Abstract

Face Recognition (FR) has advanced significantly with the development of deep learning, achieving high accuracy in several applications. However, the lack of interpretability of these systems raises concerns about their accountability, fairness, and reliability. In the present study, we propose an interactive framework to enhance the explainability of FR models by combining model-agnostic Explainable Artificial Intelligence (XAI) and Natural Language Processing (NLP) techniques. The proposed framework is able to accurately answer various questions of the user through an interactive chatbot. In particular, the explanations generated by our proposed method are in the form of natural language text and visual representations, which for example can describe how different facial regions contribute to the similarity measure between two faces. This is achieved through the automatic analysis of the output's saliency heatmaps of the face images and a BERT question-answering model, providing users with an interface that facilitates a comprehensive understanding of the FR decisions. The proposed approach is interactive, allowing the users to ask questions to get more precise information based on the user's background knowledge. More importantly, in contrast to previous studies, our solution does not decrease the face recognition performance. We demonstrate the effectiveness of the method through different experiments, highlighting its potential to make FR systems more interpretable and user-friendly, especially in sensitive applications where decision-making transparency is crucial.