A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Critic

TL;DR

This paper introduces Hamiltonian Actor-Critic (HAC), a novel model-based reinforcement learning method inspired by Pontryagin's Maximum Principle that directly optimizes a Hamiltonian to improve robustness and efficiency.

Contribution

HAC eliminates explicit value function learning in model-based RL, reducing sensitivity to model errors and providing convergence guarantees, with strong empirical performance.

Findings

HAC outperforms baselines in control performance and convergence speed.

HAC is more robust to distributional shift and out-of-distribution scenarios.

HAC matches or exceeds state-of-the-art in offline RL with limited data.

Abstract

Model-based reinforcement learning (MBRL) improves sample efficiency by leveraging learned dynamics models for policy optimization. However, the effectiveness of methods such as actor-critic is often limited by compounding model errors, which degrade long-horizon value estimation. Existing approaches, such as Model-Based Value Expansion (MVE), partially mitigate this issue through multi-step rollouts, but remain sensitive to rollout horizon selection and residual model bias. Motivated by the Pontryagin Maximum Principle (PMP), we propose Hamiltonian Actor-Critic (HAC), a model-based approach that eliminates explicit value function learning by directly optimizing a Hamiltonian defined over the learned dynamics and reward for deterministic systems. By avoiding value approximation, HAC reduces sensitivity to model errors while admitting convergence guarantees. Extensive experiments on continuous control benchmarks, in both online and offline RL settings, demonstrate that HAC outperforms model-free and MVE-based baselines in control performance, convergence speed, and robustness to distributional shift, including out-of-distribution (OOD) scenarios. In offline settings with limited data, HAC matches or exceeds state-of-the-art methods, highlighting its strong sample efficiency.