Assessing the Impact of Distribution Shift on Reinforcement Learning Performance

TL;DR

This paper introduces evaluation methods to measure the robustness of reinforcement learning algorithms under distribution shifts, emphasizing the importance of time series analysis and causal impact measurement for more reliable assessments.

Contribution

It proposes new evaluation tools for RL robustness under distribution shifts, incorporating time series analysis and causal impact measurement, addressing a gap in current reliability metrics.

Findings

Distribution shifts significantly affect RL performance.

Time series analysis provides better robustness insights.

Causal impact measurement helps understand environment effects.

Abstract

Research in machine learning is making progress in fixing its own reproducibility crisis. Reinforcement learning (RL), in particular, faces its own set of unique challenges. Comparison of point estimates, and plots that show successful convergence to the optimal policy during training, may obfuscate overfitting or dependence on the experimental setup. Although researchers in RL have proposed reliability metrics that account for uncertainty to better understand each algorithm's strengths and weaknesses, the recommendations of past work do not assume the presence of out-of-distribution observations. We propose a set of evaluation methods that measure the robustness of RL algorithms under distribution shifts. The tools presented here argue for the need to account for performance over time while the agent is acting in its environment. In particular, we recommend time series analysis as a method of observational RL evaluation. We also show that the unique properties of RL and simulated dynamic environments allow us to make stronger assumptions to justify the measurement of causal impact in our evaluations. We then apply these tools to single-agent and multi-agent environments to show the impact of introducing distribution shifts during test time. We present this methodology as a first step toward rigorous RL evaluation in the presence of distribution shifts.