Towards Human-Centred Explainability Benchmarks For Text Classification

TL;DR

This paper advocates for extending NLP text classification benchmarks to include explainability evaluation, emphasizing human-centered approaches like social media and gamification to better reflect real-world applications.

Contribution

It proposes extending existing benchmarks to evaluate explainability and grounding these in human-centered applications for more realistic assessment.

Findings

Highlights challenges in objectively evaluating explanations

Suggests using human judgments to learn explainability metrics

Emphasizes importance of real-world applicability in benchmarks

Abstract

Progress on many Natural Language Processing (NLP) tasks, such as text classification, is driven by objective, reproducible and scalable evaluation via publicly available benchmarks. However, these are not always representative of real-world scenarios where text classifiers are employed, such as sentiment analysis or misinformation detection. In this position paper, we put forward two points that aim to alleviate this problem. First, we propose to extend text classification benchmarks to evaluate the explainability of text classifiers. We review challenges associated with objectively evaluating the capabilities to produce valid explanations which leads us to the second main point: We propose to ground these benchmarks in human-centred applications, for example by using social media, gamification or to learn explainability metrics from human judgements.