Robust Adversarial Policy Optimization Under Dynamics Uncertainty

TL;DR

This paper introduces RAPO, a new robust RL framework that improves policy resilience to uncertain dynamics by combining trajectory-level adversarial rollouts with model-level reweighting, outperforming existing methods.

Contribution

It proposes a dual formulation for distributionally robust RL, integrating adversarial networks and Boltzmann reweighting to enhance stability and performance under dynamics uncertainty.

Findings

RAPO outperforms baseline robust RL methods in experiments.

The dual formulation exposes the robustness-performance trade-off.

Combining trajectory and model-level strategies improves generalization to out-of-distribution dynamics.

Abstract

Reinforcement learning (RL) policies often fail under dynamics that differ from training, a gap not fully addressed by domain randomization or existing adversarial RL methods. Distributionally robust RL provides a formal remedy but still relies on surrogate adversaries to approximate intractable primal problems, leaving blind spots that potentially cause instability and over-conservatism. We propose a dual formulation that directly exposes the robustness-performance trade-off. At the trajectory level, a temperature parameter from the dual problem is approximated with an adversarial network, yielding efficient and stable worst-case rollouts within a divergence bound. At the model level, we employ Boltzmann reweighting over dynamics ensembles, focusing on more adverse environments to the current policy rather than uniform sampling. The two components act independently and complement each other: trajectory-level steering ensures robust rollouts, while model-level sampling provides policy-sensitive coverage of adverse dynamics. The resulting framework, robust adversarial policy optimization (RAPO) outperforms robust RL baselines, improving resilience to uncertainty and generalization to out-of-distribution dynamics while maintaining dual tractability.