Direct Soft-Policy Sampling via Langevin Dynamics

TL;DR

This paper introduces Noise-Conditioned Langevin Q-Learning (NC-LQL), a novel method for directly sampling soft policies in reinforcement learning using Langevin dynamics with multi-scale noise, improving exploration and performance.

Contribution

The paper proposes NC-LQL, which integrates multi-scale noise into Langevin dynamics for efficient soft-policy sampling without explicit policy parameterization.

Findings

NC-LQL achieves competitive results on MuJoCo benchmarks.

It enables effective exploration through multi-scale noise perturbations.

The method simplifies soft-policy sampling in reinforcement learning.

Abstract

Soft policies in reinforcement learning define policies as Boltzmann distributions over state-action value functions, providing a principled mechanism for balancing exploration and exploitation. However, realizing such soft policies in practice remains challenging. Existing approaches either depend on parametric policies with limited expressivity or employ diffusion-based policies whose intractable likelihoods hinder reliable entropy estimation in soft policy objectives. We address this challenge by directly realizing soft-policy sampling via Langevin dynamics driven by the action gradient of the Q-function. This perspective leads to Langevin Q-Learning (LQL), which samples actions from the target Boltzmann distribution without explicitly parameterizing the policy. However, directly applying Langevin dynamics suffers from slow mixing in high-dimensional and non-convex Q-landscapes, limiting its practical effectiveness. To overcome this, we propose Noise-Conditioned Langevin Q-Learning (NC-LQL), which integrates multi-scale noise perturbations into the value function. NC-LQL learns a noise-conditioned Q-function that induces a sequence of progressively smoothed value landscapes, enabling sampling to transition from global exploration to precise mode refinement. On OpenAI Gym MuJoCo benchmarks, NC-LQL achieves competitive performance compared to state-of-the-art diffusion-based methods, providing a simple yet powerful solution for online RL.