Beware the Rationalization Trap! When Language Model Explainability Diverges from our Mental Models of Language

TL;DR

This paper discusses the divergence between language model explainability and human mental models, emphasizing the need for truthful, complete, and user-adaptive explanations to prevent rationalization traps.

Contribution

It introduces criteria for effective explanations and proposes a decision tree model to analyze why current methods often fail to meet these standards.

Findings

Current explanations often lack fidelity and completeness.

User-centered, adaptive explanations improve understanding.

A decision tree model illustrates explanation shortcomings.

Abstract

Language models learn and represent language differently than humans; they learn the form and not the meaning. Thus, to assess the success of language model explainability, we need to consider the impact of its divergence from a user's mental model of language. In this position paper, we argue that in order to avoid harmful rationalization and achieve truthful understanding of language models, explanation processes must satisfy three main conditions: (1) explanations have to truthfully represent the model behavior, i.e., have a high fidelity; (2) explanations must be complete, as missing information distorts the truth; and (3) explanations have to take the user's mental model into account, progressively verifying a person's knowledge and adapting their understanding. We introduce a decision tree model to showcase potential reasons why current explanations fail to reach their objectives. We further emphasize the need for human-centered design to explain the model from multiple perspectives, progressively adapting explanations to changing user expectations.