Symbolic Regression Methods for Reinforcement Learning

TL;DR

This paper introduces symbolic regression methods for reinforcement learning to generate interpretable, smooth value functions in the form of analytic expressions, outperforming neural network approaches in control tasks.

Contribution

The paper presents three novel off-line symbolic regression methods for solving the Bellman equation in reinforcement learning, providing transparent and mathematically tractable value functions.

Findings

Symbolic value functions are compact and easy to analyze.

The methods outperform neural network-based approaches in control problems.

The generated policies are well-performing and suitable for further analysis.

Abstract

Reinforcement learning algorithms can solve dynamic decision-making and optimal control problems. With continuous-valued state and input variables, reinforcement learning algorithms must rely on function approximators to represent the value function and policy mappings. Commonly used numerical approximators, such as neural networks or basis function expansions, have two main drawbacks: they are black-box models offering little insight into the mappings learned, and they require extensive trial and error tuning of their hyper-parameters. In this paper, we propose a new approach to constructing smooth value functions in the form of analytic expressions by using symbolic regression. We introduce three off-line methods for finding value functions based on a state-transition model: symbolic value iteration, symbolic policy iteration, and a direct solution of the Bellman equation. The methods are illustrated on four nonlinear control problems: velocity control under friction, one-link and two-link pendulum swing-up, and magnetic manipulation. The results show that the value functions yield well-performing policies and are compact, mathematically tractable, and easy to plug into other algorithms. This makes them potentially suitable for further analysis of the closed-loop system. A comparison with an alternative approach using neural networks shows that our method outperforms the neural network-based one.