Taming the Adversary: Stable Minimax Deep Deterministic Policy Gradient via Fractional Objectives

TL;DR

This paper introduces MMDDPG, a minimax reinforcement learning framework with a fractional objective to train disturbance-resilient policies, significantly enhancing robustness in continuous control tasks under external disturbances and uncertainties.

Contribution

It proposes a novel minimax deep deterministic policy gradient method with a fractional objective to stabilize adversarial training for robustness in RL.

Findings

Improved robustness against external force perturbations.

Enhanced performance under model parameter variations.

Effective in MuJoCo continuous control environments.

Abstract

Reinforcement learning (RL) has achieved remarkable success in a wide range of control and decision-making tasks. However, RL agents often exhibit unstable or degraded performance when deployed in environments subject to unexpected external disturbances and model uncertainties. Consequently, ensuring reliable performance under such conditions remains a critical challenge. In this paper, we propose minimax deep deterministic policy gradient (MMDDPG), a framework for learning disturbance-resilient policies in continuous control tasks. The training process is formulated as a minimax optimization problem between a user policy and an adversarial disturbance policy. In this problem, the user learns a robust policy that minimizes the objective function, while the adversary generates disturbances that maximize it. To stabilize this interaction, we introduce a fractional objective that balances task performance and disturbance magnitude. This objective prevents excessively aggressive disturbances and promotes robust learning. Experimental evaluations in MuJoCo environments demonstrate that the proposed MMDDPG achieves significantly improved robustness against both external force perturbations and model parameter variations.