Quantifying Potential Observation Missingness in Inverse Reinforcement Learning

TL;DR

This paper addresses the challenge of missing observations in behavioral datasets used for inverse reinforcement learning, proposing a method to quantify how such missing data can affect inferred rewards.

Contribution

It introduces a practical algorithm to identify minimal observation perturbations that make expert actions appear optimal, highlighting the impact of missing data on IRL.

Findings

The algorithm effectively quantifies potential missing observations in synthetic and real datasets.

Missing data can significantly influence the reward functions inferred by IRL.

The method provides insights into the robustness of IRL in real-world applications.

Abstract

Inverse reinforcement learning (IRL), which infers reward functions from demonstrations, is a valuable tool for modeling and understanding decision-making behavior. Many variants of IRL have been developed to capture complexities of human decision-making, such as subjective beliefs, imperfect planning, and dynamic goals. However, an often-overlooked issue in real-world behavioral datasets is that the recorded data may be missing observations that were available to the original decision-maker. In use-inspired settings such as healthcare, this can make expert actions appear suboptimal, even when they were near-optimal given the information available at the time. As a result, the rewards learned by standard IRL may be misleading. In this paper, we identify the minimal perturbations to the recorded observations needed for the expert's actions to appear optimal. We develop a practical algorithm for this problem and demonstrate its utility for quantifying the possible extent of missing observations in behavioral datasets through extensive experiments on synthetic navigation tasks, a cancer treatment simulator, and ICU treatment data.