The Irrationality of Neural Rationale Models

TL;DR

Neural rationale models, while popular for interpretability in NLP, may not be truly rational or interpretable, highlighting the need for more rigorous evaluation methods.

Contribution

This paper challenges the assumption that rationale models are inherently interpretable, providing philosophical and empirical evidence to question their rationality.

Findings

Rationale models may be less rational than assumed

Empirical evidence questions the interpretability of rationale models

Calls for more rigorous evaluation of interpretability properties

Abstract

Neural rationale models are popular for interpretable predictions of NLP tasks. In these, a selector extracts segments of the input text, called rationales, and passes these segments to a classifier for prediction. Since the rationale is the only information accessible to the classifier, it is plausibly defined as the explanation. Is such a characterization unconditionally correct? In this paper, we argue to the contrary, with both philosophical perspectives and empirical evidence suggesting that rationale models are, perhaps, less rational and interpretable than expected. We call for more rigorous and comprehensive evaluations of these models to ensure desired properties of interpretability are indeed achieved. The code can be found at https://github.com/yimingz89/Neural-Rationale-Analysis.