Task-Specific Embeddings for Ante-Hoc Explainable Text Classification

TL;DR

This paper introduces task-specific embeddings for text classification that enable interpretable, ante-hoc explanations and incremental learning without sacrificing accuracy, by replacing softmax classifiers with a distance-based approach.

Contribution

It proposes a novel training objective for learning class-specific embeddings that support explainability and incremental learning in text classification.

Findings

Embeddings allow direct explanation of classification decisions.

Qualitative analysis helps identify data quality issues.

Distances generalize to unseen classes, enabling incremental learning.

Abstract

Current state-of-the-art approaches to text classification typically leverage BERT-style Transformer models with a softmax classifier, jointly fine-tuned to predict class labels of a target task. In this paper, we instead propose an alternative training objective in which we learn task-specific embeddings of text: our proposed objective learns embeddings such that all texts that share the same target class label should be close together in the embedding space, while all others should be far apart. This allows us to replace the softmax classifier with a more interpretable k-nearest-neighbor classification approach. In a series of experiments, we show that this yields a number of interesting benefits: (1) The resulting order induced by distances in the embedding space can be used to directly explain classification decisions. (2) This facilitates qualitative inspection of the training data, helping us to better understand the problem space and identify labelling quality issues. (3) The learned distances to some degree generalize to unseen classes, allowing us to incrementally add new classes without retraining the model. We present extensive experiments which show that the benefits of ante-hoc explainability and incremental learning come at no cost in overall classification accuracy, thus pointing to practical applicability of our proposed approach.