Order-sensitive Shapley Values for Evaluating Conceptual Soundness of NLP Models

TL;DR

This paper introduces Order-sensitive Shapley Values (OSV), a new explanation method for sequential data, to evaluate whether NLP models truly learn word order concepts, revealing limitations in current models.

Contribution

The paper proposes OSV, a novel explanation technique for sequential data, and demonstrates its effectiveness in assessing models' understanding of word order across various NLP tasks.

Findings

OSV is more faithful than gradient-based methods in explaining model behavior.

BERT-based NLI models rely on word presence, not order.

Some sentiment models fail to learn negation properly.

Abstract

Previous works show that deep NLP models are not always conceptually sound: they do not always learn the correct linguistic concepts. Specifically, they can be insensitive to word order. In order to systematically evaluate models for their conceptual soundness with respect to word order, we introduce a new explanation method for sequential data: Order-sensitive Shapley Values (OSV). We conduct an extensive empirical evaluation to validate the method and surface how well various deep NLP models learn word order. Using synthetic data, we first show that OSV is more faithful in explaining model behavior than gradient-based methods. Second, applying to the HANS dataset, we discover that the BERT-based NLI model uses only the word occurrences without word orders. Although simple data augmentation improves accuracy on HANS, OSV shows that the augmented model does not fundamentally improve the model's learning of order. Third, we discover that not all sentiment analysis models learn negation properly: some fail to capture the correct syntax of the negation construct. Finally, we show that pretrained language models such as BERT may rely on the absolute positions of subject words to learn long-range Subject-Verb Agreement. With each NLP task, we also demonstrate how OSV can be leveraged to generate adversarial examples.