Model-based Reinforcement Learning with a Hamiltonian Canonical ODE Network

TL;DR

This paper introduces NODA, a neural ODE auto-encoder leveraging Hamiltonian mechanics to improve sample efficiency and physical plausibility in model-based reinforcement learning for complex environments.

Contribution

The paper proposes NODA, a novel neural ODE auto-encoder incorporating Hamiltonian mechanics, enhancing efficiency and physical consistency in environment modeling for RL.

Findings

NODA effectively models environment dynamics with high sample efficiency.

NODA provides theoretical bounds for multi-step transition and value errors.

Experiments demonstrate improved early-stage RL performance with NODA.

Abstract

Model-based reinforcement learning usually suffers from a high sample complexity in training the world model, especially for the environments with complex dynamics. To make the training for general physical environments more efficient, we introduce Hamiltonian canonical ordinary differential equations into the learning process, which inspires a novel model of neural ordinary differential auto-encoder (NODA). NODA can model the physical world by nature and is flexible to impose Hamiltonian mechanics (e.g., the dimension of the physical equations) which can further accelerate training of the environment models. It can consequentially empower an RL agent with the robust extrapolation using a small amount of samples as well as the guarantee on the physical plausibility. Theoretically, we prove that NODA has uniform bounds for multi-step transition errors and value errors under certain conditions. Extensive experiments show that NODA can learn the environment dynamics effectively with a high sample efficiency, making it possible to facilitate reinforcement learning agents at the early stage.