Distributional value gradients for stochastic environments

TL;DR

This paper introduces Distributional Sobolev Training, a novel method extending distributional reinforcement learning to model value gradients in stochastic environments, improving sample efficiency and robustness.

Contribution

It extends distributional RL to model value gradients using Sobolev spaces, with a new theoretical framework and practical implementation via cVAE and MSMMD.

Findings

Effective on stochastic toy problems

Outperforms baselines on MuJoCo benchmarks

Proves contraction of Sobolev Bellman operator

Abstract

Gradient-regularized value learning methods improve sample efficiency by leveraging learned models of transition dynamics and rewards to estimate return gradients. However, existing approaches, such as MAGE, struggle in stochastic or noisy environments, limiting their applicability. In this work, we address these limitations by extending distributional reinforcement learning on continuous state-action spaces to model not only the distribution over scalar state-action value functions but also over their gradients. We refer to this approach as Distributional Sobolev Training. Inspired by Stochastic Value Gradients (SVG), our method utilizes a one-step world model of reward and transition distributions implemented via a conditional Variational Autoencoder (cVAE). The proposed framework is sample-based and employs Max-sliced Maximum Mean Discrepancy (MSMMD) to instantiate the distributional Bellman operator. We prove that the Sobolev-augmented Bellman operator is a contraction with a unique fixed point, and highlight a fundamental smoothness trade-off underlying contraction in gradient-aware RL. To validate our method, we first showcase its effectiveness on a simple stochastic reinforcement learning toy problem, then benchmark its performance on several MuJoCo environments.