Model-based Policy Optimization using Symbolic World Model

TL;DR

This paper introduces a symbolic world model for model-based policy optimization in robotics, improving sample efficiency by using symbolic regression to approximate dynamics with fewer parameters, leading to better extrapolation.

Contribution

It proposes a novel symbolic dynamics model for policy optimization, enhancing sample efficiency and accuracy over neural network models in robotic control tasks.

Findings

Achieves higher sample efficiency than baseline methods

Uses fewer parameters for dynamics approximation

Demonstrates superior performance in simulated tasks

Abstract

The application of learning-based control methods in robotics presents significant challenges. One is that model-free reinforcement learning algorithms use observation data with low sample efficiency. To address this challenge, a prevalent approach is model-based reinforcement learning, which involves employing an environment dynamics model. We suggest approximating transition dynamics with symbolic expressions, which are generated via symbolic regression. Approximation of a mechanical system with a symbolic model has fewer parameters than approximation with neural networks, which can potentially lead to higher accuracy and quality of extrapolation. We use a symbolic dynamics model to generate trajectories in model-based policy optimization to improve the sample efficiency of the learning algorithm. We evaluate our approach across various tasks within simulated environments. Our method demonstrates superior sample efficiency in these tasks compared to model-free and model-based baseline methods.