Measuring Interventional Robustness in Reinforcement Learning

TL;DR

This paper introduces a new measure called interventional robustness (IR) to evaluate how consistent reinforcement learning policies are under different incidental training conditions, revealing that high performance does not always mean high IR.

Contribution

The paper defines and quantifies interventional robustness in reinforcement learning, providing a new way to assess policy stability across various training interventions.

Findings

IR varies with training amount and algorithm type

High performance does not necessarily imply high IR

IR measurement applied to eight algorithms in Atari environments

Abstract

Recent work in reinforcement learning has focused on several characteristics of learned policies that go beyond maximizing reward. These properties include fairness, explainability, generalization, and robustness. In this paper, we define interventional robustness (IR), a measure of how much variability is introduced into learned policies by incidental aspects of the training procedure, such as the order of training data or the particular exploratory actions taken by agents. A training procedure has high IR when the agents it produces take very similar actions under intervention, despite variation in these incidental aspects of the training procedure. We develop an intuitive, quantitative measure of IR and calculate it for eight algorithms in three Atari environments across dozens of interventions and states. From these experiments, we find that IR varies with the amount of training and type of algorithm and that high performance does not imply high IR, as one might expect.