Co-constructing Explanations for AI Systems using Provenance

TL;DR

This paper proposes an interactive agent that collaborates with users to generate explanations of AI systems by leveraging data provenance, aiming to improve interpretability and user understanding.

Contribution

It introduces a novel interactive approach for explanation generation that combines provenance data with user collaboration, supported by a prototype and a scalable evaluation framework.

Findings

Prototype demonstrates interactive explanation capabilities.

Evaluation framework uses user simulations and large language models.

Initial results show promise for user-centered explanations.

Abstract

Modern AI systems are complex workflows containing multiple components and data sources. Data provenance provides the ability to interrogate and potentially explain the outputs of these systems. However, provenance is often too detailed and not contextualized for the user trying to understand the AI system. In this work, we present our vision for an interactive agent that works together with the user to co-construct an explanation that is simultaneously useful to the user as well as grounded in data provenance. To illustrate this vision, we present: 1) an initial prototype of such an agent; and 2) a scalable evaluation framework based on user simulations and a large language model as a judge approach.