On the Tension Between Optimality and Adversarial Robustness in Policy Optimization

TL;DR

This paper explores the practical conflict between optimality and robustness in deep reinforcement learning, revealing how adversarial training reshapes the landscape and proposing a bilevel framework to balance these goals effectively.

Contribution

It introduces BARPO, a bilevel policy optimization framework that modulates adversary strength to reconcile the theoretical alignment of optimality and robustness in practice.

Findings

BARPO outperforms standard adversarially robust policy optimization.

Adversary strength modulation improves navigability and robustness.

Tradeoff between robustness and optimality is linked to landscape reshaping by adversaries.

Abstract

Achieving optimality and adversarial robustness in deep reinforcement learning has long been regarded as conflicting goals. Nonetheless, recent theoretical insights presented in CAR suggest a potential alignment, raising the important question of how to realize this in practice. This paper first identifies a key gap between theory and practice by comparing standard policy optimization (SPO) and adversarially robust policy optimization (ARPO). Although they share theoretical consistency, a fundamental tension between robustness and optimality arises in practical policy gradient methods. SPO tends toward convergence to vulnerable first-order stationary policies (FOSPs) with strong natural performance, whereas ARPO typically favors more robust FOSPs at the expense of reduced returns. Furthermore, we attribute this tradeoff to the reshaping effect of the strongest adversary in ARPO, which significantly complicates the global landscape by inducing deceptive sticky FOSPs. This improves robustness but makes navigation more challenging. To alleviate this, we develop the BARPO, a bilevel framework unifying SPO and ARPO by modulating adversary strength, thereby facilitating navigability while preserving global optima. Extensive empirical results demonstrate that BARPO consistently outperforms vanilla ARPO, providing a practical approach to reconcile theoretical and empirical performance.