Manipulating and Measuring Model Interpretability

TL;DR

This study experimentally investigates how different factors like feature count and transparency affect model interpretability, revealing that clearer models improve simulation but may hinder mistake detection due to information overload.

Contribution

It provides empirical evidence that transparency and feature count influence interpretability, challenging assumptions about what makes models more understandable.

Findings

Clear models improve prediction simulation

Transparency can reduce mistake detection ability

More features do not necessarily enhance interpretability

Abstract

With machine learning models being increasingly used to aid decision making even in high-stakes domains, there has been a growing interest in developing interpretable models. Although many supposedly interpretable models have been proposed, there have been relatively few experimental studies investigating whether these models achieve their intended effects, such as making people more closely follow a model's predictions when it is beneficial for them to do so or enabling them to detect when a model has made a mistake. We present a sequence of pre-registered experiments (N=3,800) in which we showed participants functionally identical models that varied only in two factors commonly thought to make machine learning models more or less interpretable: the number of features and the transparency of the model (i.e., whether the model internals are clear or black box). Predictably, participants who saw a clear model with few features could better simulate the model's predictions. However, we did not find that participants more closely followed its predictions. Furthermore, showing participants a clear model meant that they were less able to detect and correct for the model's sizable mistakes, seemingly due to information overload. These counterintuitive findings emphasize the importance of testing over intuition when developing interpretable models.