Am I Building a White Box Agent or Interpreting a Black Box Agent?

TL;DR

This paper discusses the fidelity-accuracy dilemma in explainable AI, emphasizing its importance in dynamic environments and clarifying the distinct research directions of building white box agents versus interpreting black box agents.

Contribution

It highlights the distinction between constructing transparent agents and interpreting opaque ones, stressing the need for clarity in agent interpretability research.

Findings

Fidelity-accuracy trade-off is crucial in explainable AI.

Building white box agents and interpreting black box agents are separate research paths.

Misconflating these paths can hinder progress in agent interpretability.

Abstract

The rule extraction literature contains the notion of a fidelity-accuracy dilemma: when building an interpretable model of a black box function, optimising for fidelity is likely to reduce performance on the underlying task, and vice versa. I reassert the relevance of this dilemma for the modern field of explainable artificial intelligence, and highlight how it is compounded when the black box is an agent interacting with a dynamic environment. I then discuss two independent research directions - building white box agents and interpreting black box agents - which are both coherent and worthy of attention, but must not be conflated by researchers embarking on projects in the domain of agent interpretability.