Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems

TL;DR

This paper proposes a role-based model to analyze interpretability in machine learning systems, emphasizing that interpretability depends on the agent's role and context, which impacts how systems should be designed and evaluated.

Contribution

It introduces a novel role-based framework for understanding interpretability, highlighting the importance of agent-specific perspectives in interpretability assessments.

Findings

The model clarifies how different agent roles influence interpretability goals.

Illustrations demonstrate the model's application across various scenarios.

Suggestions for practical use in research, development, and regulation.

Abstract

Several researchers have argued that a machine learning system's interpretability should be defined in relation to a specific agent or task: we should not ask if the system is interpretable, but to whom is it interpretable. We describe a model intended to help answer this question, by identifying different roles that agents can fulfill in relation to the machine learning system. We illustrate the use of our model in a variety of scenarios, exploring how an agent's role influences its goals, and the implications for defining interpretability. Finally, we make suggestions for how our model could be useful to interpretability researchers, system developers, and regulatory bodies auditing machine learning systems.