Adversarial Latent-State Training for Robust Policies in Partially Observable Domains

TL;DR

This paper introduces a theoretical framework and empirical validation for improving the robustness of reinforcement learning policies under adversarially shifted hidden initial states in partially observable environments.

Contribution

It formalizes the adversarial latent-initial-state POMDP setting, proves a minimax principle, and provides practical diagnostics and algorithms for robustness enhancement.

Findings

Targeted training reduces robustness gaps from 10.3 to 3.1 shots.

Theoretical diagnostics align with empirical results.

Framework offers a clear evaluation game and insights into implementation limits.

Abstract

Robustness under latent distribution shift remains challenging in partially observable reinforcement learning. We formalize a focused setting where an adversary selects a hidden initial latent distribution before the episode, termed an adversarial latent-initial-state POMDP. Theoretically, we prove a latent minimax principle, characterize worst-case defender distributions, and derive approximate best-response inequalities with finite-sample concentration bounds that make the optimization and sampling terms explicit. Empirically, using a Battleship benchmark, we demonstrate that targeted exposure to shifted latent distributions reduces average robustness gaps between Spread and Uniform distributions from 10.3 to 3.1 shots at equal budget. Furthermore, iterative best-response training exhibits budget-sensitive behavior that is qualitatively consistent with the theorem-guided diagnostics once one accounts for discounted PPO surrogates and finite-sample noise. Ultimately, we show that for latent-initial-state problems, the framework yields a clean evaluation game and useful theorem-motivated diagnostics while also making clear where implementation-level surrogates and optimization limits enter.