To what extent do human explanations of model behavior align with actual model behavior?

TL;DR

This study assesses how well human explanations of NLP model decisions align with actual model behavior, revealing that smaller models often align better with human explanations than larger ones, and that alignment is independent of model accuracy.

Contribution

The paper introduces three metrics to quantify alignment between human explanations and model behavior, and evaluates multiple models to analyze factors affecting this alignment.

Findings

BERT-base shows highest alignment with human explanations.

Smaller transformer models tend to align better than larger ones.

Alignment with human explanations is independent of model accuracy.

Abstract

Given the increasingly prominent role NLP models (will) play in our lives, it is important for human expectations of model behavior to align with actual model behavior. Using Natural Language Inference (NLI) as a case study, we investigate the extent to which human-generated explanations of models' inference decisions align with how models actually make these decisions. More specifically, we define three alignment metrics that quantify how well natural language explanations align with model sensitivity to input words, as measured by integrated gradients. Then, we evaluate eight different models (the base and large versions of BERT, RoBERTa and ELECTRA, as well as anRNN and bag-of-words model), and find that the BERT-base model has the highest alignment with human-generated explanations, for all alignment metrics. Focusing in on transformers, we find that the base versions tend to have higher alignment with human-generated explanations than their larger counterparts, suggesting that increasing the number of model parameters leads, in some cases, to worse alignment with human explanations. Finally, we find that a model's alignment with human explanations is not predicted by the model's accuracy, suggesting that accuracy and alignment are complementary ways to evaluate models.