A Bayesian Approach to Identifying Representational Errors

TL;DR

This paper introduces GEM, a Bayesian generative model that identifies whether errors in AI systems or humans stem from representational limitations or other factors, aiding targeted improvements.

Contribution

The work presents a novel Bayesian inference method for GEM that effectively distinguishes between representational and non-representational errors in diverse domains.

Findings

Successfully recovers blind spots in reinforcement learning agents

Effectively identifies representational errors in human users

Demonstrates utility across multiple domains

Abstract

Trained AI systems and expert decision makers can make errors that are often difficult to identify and understand. Determining the root cause for these errors can improve future decisions. This work presents Generative Error Model (GEM), a generative model for inferring representational errors based on observations of an actor's behavior (either simulated agent, robot, or human). The model considers two sources of error: those that occur due to representational limitations -- "blind spots" -- and non-representational errors, such as those caused by noise in execution or systematic errors present in the actor's policy. Disambiguating these two error types allows for targeted refinement of the actor's policy (i.e., representational errors require perceptual augmentation, while other errors can be reduced through methods such as improved training or attention support). We present a Bayesian inference algorithm for GEM and evaluate its utility in recovering representational errors on multiple domains. Results show that our approach can recover blind spots of both reinforcement learning agents as well as human users.