Delightful Distributed Policy Gradient

TL;DR

The paper introduces Delightful Policy Gradient (DG), a novel method for distributed reinforcement learning that effectively handles high-surprisal data by gating updates with delight, improving sample efficiency and robustness.

Contribution

DG separates failures and successes using advantage and surprisal, suppressing failures without losing successes, and demonstrates superior performance in complex, real-world tasks.

Findings

DG outperforms importance-weighted PG on MNIST with staleness.

DG achieves nearly order-of-magnitude lower error on complex sequence tasks.

Sample efficiency of DG grows with task complexity.

Abstract

Distributed reinforcement learning trains on data from stale, buggy, or mismatched actors, producing actions with high surprisal (negative log-probability) under the learner's policy. The core difficulty is not surprising data per se, but \emph{negative learning from surprising data}. High-surprisal failures can dominate finite-batch updates through large perpendicular components, while high-surprisal successes reveal opportunities the current policy would otherwise miss. The \textit{Delightful Policy Gradient} (DG) separates these cases by gating each update with delight, the product of advantage and surprisal, suppressing rare failures and preserving rare successes without behavior probabilities. In a tabular analysis, DG suppresses the perpendicular second moment of high-surprisal failures by a policy-overlap factor that vanishes as the learner improves. The advantage sign is essential for surprisal-based filtering: any learner-probability-only gate that suppresses rare failures also suppresses rare successes. On MNIST with simulated staleness, DG without off-policy correction outperforms importance-weighted PG with exact behavior probabilities. On a transformer sequence task with staleness, actor bugs, reward corruption, and rare discovery, DG often achieves nearly order-of-magnitude lower error. When all four frictions act simultaneously, its sample-efficiency advantage is order-of-magnitude and grows with task complexity.