"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification

TL;DR

This paper introduces a protocol for evaluating the faithfulness of input salience methods in text classification, revealing that popular methods often perform poorly in identifying model shortcuts, and emphasizes the need for task-specific evaluation.

Contribution

The paper proposes a novel protocol using synthetic data for ground truth in faithfulness evaluation of salience methods in text classification.

Findings

Some popular salience methods perform poorly in identifying shortcuts.

The protocol helps determine the most faithful attribution method for a given task.

Evaluation reveals significant differences among salience methods on various datasets.

Abstract

Feature attribution a.k.a. input salience methods which assign an importance score to a feature are abundant but may produce surprisingly different results for the same model on the same input. While differences are expected if disparate definitions of importance are assumed, most methods claim to provide faithful attributions and point at the features most relevant for a model's prediction. Existing work on faithfulness evaluation is not conclusive and does not provide a clear answer as to how different methods are to be compared. Focusing on text classification and the model debugging scenario, our main contribution is a protocol for faithfulness evaluation that makes use of partially synthetic data to obtain ground truth for feature importance ranking. Following the protocol, we do an in-depth analysis of four standard salience method classes on a range of datasets and shortcuts for BERT and LSTM models and demonstrate that some of the most popular method configurations provide poor results even for simplest shortcuts. We recommend following the protocol for each new task and model combination to find the best method for identifying shortcuts.