Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learning

TL;DR

This paper demonstrates that physics-based models, when the mechanical structure is known, outperform black-box models in offline model-based reinforcement learning for real-world tasks, enabling effective learning with minimal data.

Contribution

It introduces a physics-based modeling approach for offline MBRL that leverages known mechanical structures and automatic differentiation, showing superior performance over black-box models in a real-world task.

Findings

Physics-based models outperform black-box models in the BiC task.

Black-box models produce diverging trajectories and unviable policies.

Physics models learn effectively with only 4 minutes of data.

Abstract

A limitation of model-based reinforcement learning (MBRL) is the exploitation of errors in the learned models. Black-box models can fit complex dynamics with high fidelity, but their behavior is undefined outside of the data distribution.Physics-based models are better at extrapolating, due to the general validity of their informed structure, but underfit in the real world due to the presence of unmodeled phenomena. In this work, we demonstrate experimentally that for the offline model-based reinforcement learning setting, physics-based models can be beneficial compared to high-capacity function approximators if the mechanical structure is known. Physics-based models can learn to perform the ball in a cup (BiC) task on a physical manipulator using only 4 minutes of sampled data using offline MBRL. We find that black-box models consistently produce unviable policies for BiC as all predicted trajectories diverge to physically impossible state, despite having access to more data than the physics-based model. In addition, we generalize the approach of physics parameter identification from modeling holonomic multi-body systems to systems with nonholonomic dynamics using end-to-end automatic differentiation. Videos: https://sites.google.com/view/ball-in-a-cup-in-4-minutes/