Markov Decision Processes of the Third Kind: Learning Distributions by Policy Gradient Descent

TL;DR

This paper introduces a novel policy-gradient method for distributional Markov Decision Processes, enabling learning policies that steer reward distributions toward specific targets, with proven convergence and demonstrated effectiveness in complex distribution matching.

Contribution

It proposes a new model-free policy-gradient algorithm for distributional MDPs using neural networks and characteristic-function loss, with convergence guarantees and practical validation.

Findings

Successfully matches complex target distributions.

Recovers classical optimal policies in known cases.

Reveals non-uniqueness phenomena in distributional control.

Abstract

The goal of this paper is to analyze distributional Markov Decision Processes as a class of control problems in which the objective is to learn policies that steer the distribution of a cumulative reward toward a prescribed target law, rather than optimizing an expected value or a risk functional. To solve the resulting distributional control problem in a model-free setting, we propose a policy-gradient algorithm based on neural-network parameterizations of randomized Markov policies, defined on an augmented state space and a sample-based evaluation of the characteristic-function loss. Under mild regularity and growth assumptions, we prove convergence of the algorithm to stationary points using stochastic approximation techniques. Several numerical experiments illustrate the ability of the method to match complex target distributions, recover classical optimal policies when they exist, and reveal intrinsic non-uniqueness phenomena specific to distributional control.