Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behavior

TL;DR

Shared Interest introduces quantitative metrics to compare neural network saliency with human reasoning, enabling large-scale analysis of model behavior and identification of recurring patterns to assess trustworthiness.

Contribution

The paper presents Shared Interest, a novel set of metrics for systematically comparing model saliency with human annotations, facilitating large-scale analysis of model reasoning patterns.

Findings

Identified eight recurring model behavior patterns.

Demonstrated how Shared Interest can assess model trustworthiness.

Showed that Shared Interest uncovers issues missed by manual analysis.

Abstract

Saliency methods -- techniques to identify the importance of input features on a model's output -- are a common step in understanding neural network behavior. However, interpreting saliency requires tedious manual inspection to identify and aggregate patterns in model behavior, resulting in ad hoc or cherry-picked analysis. To address these concerns, we present Shared Interest: metrics for comparing model reasoning (via saliency) to human reasoning (via ground truth annotations). By providing quantitative descriptors, Shared Interest enables ranking, sorting, and aggregating inputs, thereby facilitating large-scale systematic analysis of model behavior. We use Shared Interest to identify eight recurring patterns in model behavior, such as cases where contextual features or a subset of ground truth features are most important to the model. Working with representative real-world users, we show how Shared Interest can be used to decide if a model is trustworthy, uncover issues missed in manual analyses, and enable interactive probing.