Understanding Behavioral Metric Learning: A Large-Scale Study on Distracting Reinforcement Learning Environments

TL;DR

This study systematically evaluates recent behavioral metric learning methods in deep reinforcement learning across diverse noisy environments, highlighting their impact on robustness and proposing new evaluation metrics and an open-source codebase.

Contribution

It provides a comprehensive benchmark of five recent metric learning approaches, introduces a denoising factor for evaluation, and isolates metric estimation effects with a new experimental setting.

Findings

Metrics improve robustness to noise in RL environments.

Isolated metric estimation clarifies the contribution of metric learning.

Open-source codebase facilitates future research.

Abstract

A key approach to state abstraction is approximating behavioral metrics (notably, bisimulation metrics) in the observation space and embedding these learned distances in the representation space. While promising for robustness to task-irrelevant noise, as shown in prior work, accurately estimating these metrics remains challenging, requiring various design choices that create gaps between theory and practice. Prior evaluations focus mainly on final returns, leaving the quality of learned metrics and the source of performance gains unclear. To systematically assess how metric learning works in deep reinforcement learning (RL), we evaluate five recent approaches, unified conceptually as isometric embeddings with varying design choices. We benchmark them with baselines across 20 state-based and 14 pixel-based tasks, spanning 370 task configurations with diverse noise settings. Beyond final returns, we introduce the evaluation of a denoising factor to quantify the encoder's ability to filter distractions. To further isolate the effect of metric learning, we propose and evaluate an isolated metric estimation setting, in which the encoder is influenced solely by the metric loss. Finally, we release an open-source, modular codebase to improve reproducibility and support future research on metric learning in deep RL.