Quotient-Categorical Representations for Bellman-Compatible Average-Reward Distributional Reinforcement Learning

TL;DR

This paper introduces a quotient-space formulation for average-reward distributional reinforcement learning, addressing the ill-posedness of direct distributional analogues on the real line and establishing convergence properties.

Contribution

It proposes a quotient-categorical approach that respects symmetry in average-reward RL, with fixed points and convergence guarantees for the associated operators and recursions.

Findings

The quotient-space formulation is well-defined and non-expansive.

Sampled recursions converge almost surely under certain conditions.

Gain estimation can be integrated with convergence guarantees.

Abstract

Average-reward reinforcement learning requires estimating the gain and the bias, which is defined only up to an additive constant. This makes direct distributional analogues ill-posed on the real line. We introduce a quotient-space formulation in which state-indexed bias laws are identified up to a common translation, together with a categorical parameterization that respects this symmetry. On this quotient-categorical space, we define a projected average-reward distributional operator and show that it is well-defined, non-expansive in a coordinate Cram\'er metric, and admits fixed points. We then study sampled recursions whose mean-field maps are asynchronous relaxations of this operator. In an idealized centered-reward setting, a one-state temporal-difference update enjoys almost sure convergence together with finite-iteration residual bounds under both i.i.d. and Markovian sampling. When the gain is unknown, we augment the recursion with an online gain estimator, and prove non-expansiveness and Markovian convergence of the resulting coupled scheme. Finally, we show that synchronous exact updates are gain-independent at the quotient-law level, isolating a structural contrast between ideal quotient distributions and practical fixed-grid categorical representations.