RN-D: Discretized Categorical Actors with Regularized Networks for On-Policy Reinforcement Learning

TL;DR

This paper introduces discretized categorical actors with regularized networks for on-policy reinforcement learning, leading to improved stability and state-of-the-art performance in continuous control tasks.

Contribution

It proposes a novel discretized policy representation and regularized actor networks, enhancing on-policy RL stability and performance over traditional Gaussian policies.

Findings

Achieves state-of-the-art results on continuous-control benchmarks.

Discretized categorical actors outperform Gaussian actors in stability.

Regularized networks improve policy optimization robustness.

Abstract

On-policy deep reinforcement learning remains a dominant paradigm for continuous control, yet standard implementations rely on Gaussian actors and relatively shallow MLP policies, often leading to brittle optimization when gradients are noisy and policy updates must be conservative. In this paper, we revisit policy representation as a first-class design choice for on-policy optimization. We study discretized categorical actors that represent each action dimension with a distribution over bins, yielding a policy objective that resembles a cross-entropy loss. Building on architectural advances from supervised learning, we further propose regularized actor networks, while keeping critic design fixed. Our results show that simply replacing the standard actor network with our discretized regularized actor yields consistent gains and achieve the state-of-the-art performance across diverse continuous-control benchmarks.