Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting

TL;DR

This paper introduces PG-DPO, a new framework for reinforcement learning with non-exponential discounting, overcoming Bellman recursion limitations by integrating Pontryagin's principle with Monte Carlo methods.

Contribution

It presents a novel variational approach that replaces traditional Bellman recursion, enabling stable and accurate learning under non-exponential discounting.

Findings

PG-DPO outperforms traditional methods on hyperbolic and survival-discount benchmarks.

It improves stability and accuracy where standard solvers and baselines fail.

The framework effectively handles non-exponential discounting in reinforcement learning.

Abstract

Most value-based and actor--critic reinforcement learning methods rely on Bellman-style recursions, yet these recursions collapse under non-exponential discounting common in human preferences and survival processes. We show the breakdown is structural: exponential discounting sits at a fragile intersection of multiplicativity and time homogeneity, and violating either property breaks standard dynamic programming. To overcome this, we propose Pontryagin-Guided Direct Policy Optimization (PG-DPO), a variational framework that abandons recursion and couples the Pontryagin Maximum Principle with Monte Carlo rollouts via an Adjoint-MC projection enforcing pointwise Hamiltonian maximization. Across multi-dimensional hyperbolic and survival-discount benchmarks, PG-DPO improves accuracy and stability where equation-driven solvers and critic-based baselines diverge.