Physics Informed Viscous Value Representations

TL;DR

This paper introduces a physics-informed regularization for offline goal-conditioned reinforcement learning, grounded in the viscosity solution of the HJB equation, improving value estimation in complex, high-dimensional environments.

Contribution

It proposes a novel regularization based on the viscosity solution of the HJB equation, leveraging the Feynman-Kac theorem for stable Monte Carlo estimation in high-dimensional settings.

Findings

Enhanced geometric consistency in navigation tasks

Improved value estimation in complex manipulation environments

Broad applicability demonstrated across tasks

Abstract

Offline goal-conditioned reinforcement learning (GCRL) learns goal-conditioned policies from static pre-collected datasets. However, accurate value estimation remains a challenge due to the limited coverage of the state-action space. Recent physics-informed approaches have sought to address this by imposing physical and geometric constraints on the value function through regularization defined over first-order partial differential equations (PDEs), such as the Eikonal equation. However, these formulations can often be ill-posed in complex, high-dimensional environments. In this work, we propose a physics-informed regularization derived from the viscosity solution of the Hamilton-Jacobi-Bellman (HJB) equation. By providing a physics-based inductive bias, our approach grounds the learning process in optimal control theory, explicitly regularizing and bounding updates during value iterations. Furthermore, we leverage the Feynman-Kac theorem to recast the PDE solution as an expectation, enabling a tractable Monte Carlo estimation of the objective that avoids numerical instability in higher-order gradients. Experiments demonstrate that our method improves geometric consistency, making it broadly applicable to navigation and high-dimensional, complex manipulation tasks. Open-source codes are available at https://github.com/HrishikeshVish/phys-fk-value-GCRL.