Lipschitz-Regularized Critics Lead to Policy Robustness Against Transition Dynamics Uncertainty

TL;DR

This paper introduces PPO-PGDLC, a reinforcement learning algorithm that combines Lipschitz-regularized critics with adversarial state sampling to enhance policy robustness against transition uncertainties, validated on control and robotic tasks.

Contribution

It proposes a novel RL method integrating Lipschitz regularization with adversarial state sampling, addressing critic-only regularization impact and real-world robustness validation.

Findings

PPO-PGDLC outperforms baseline algorithms in control and robotic tasks.

The method produces smoother actions under environmental perturbations.

Experimental results demonstrate improved robustness and performance.

Abstract

Uncertainties in transition dynamics pose a critical challenge in reinforcement learning (RL), often resulting in performance degradation of trained policies when deployed on hardware. Many robust RL approaches follow two strategies: enforcing smoothness in actor or actor-critic modules with Lipschitz regularization, or learning robust Bellman operators. However, the first strategy does not investigate the impact of critic-only Lipschitz regularization on policy robustness, while the second lacks comprehensive validation in real-world scenarios. Building on this gap and prior work, we propose PPO-PGDLC, an algorithm based on Proximal Policy Optimization (PPO) that integrates Projected Gradient Descent (PGD) with a Lipschitz-regularized critic (LC). The PGD component calculates the adversarial state within an uncertainty set to approximate the robust Bellman operator, and the Lipschitz-regularized critic further improves the smoothness of learned policies. Experimental results on two classic control tasks and one real-world robotic locomotion task demonstrates that, compared to several baseline algorithms, PPO-PGDLC achieves better performance and predicts smoother actions under environmental perturbations.