To Learn or Not to Learn: A Litmus Test for Using Reinforcement Learning in Control

TL;DR

This paper introduces a simulation-based litmus test to determine whether reinforcement learning offers advantages over classical control methods without the need for RL training.

Contribution

The authors propose a novel, efficient method to assess the potential benefits of RL in control problems by analyzing model uncertainties and their impact.

Findings

The test accurately predicts when RL outperforms model-based control.

It reduces computational costs by avoiding unnecessary RL training.

The method is applicable to various control benchmarks.

Abstract

Reinforcement learning (RL) can be a powerful alternative to classical control methods when standard model-based control is insufficient, e.g., when deriving a suitable model is intractable or impossible. In many cases, however, the choice between model-based and RL-based control is not obvious. Due to the high computational costs of training RL agents, RL-based control should be limited to cases where it is expected to yield superior results compared to model-based control. To the best of our knowledge, there exists no approach to quantify the benefit of RL-based control that does not require RL training. In this work, we present a computationally efficient, purely simulation-based litmus test predicting whether RL-based control is superior to model-based control. Our test evaluates the suitability of the given model for model-based control by analyzing the impact of model uncertainties on the control problem. For this, we use reachset-conformant model identification combined with simulation-based analysis. This is followed by a learnability evaluation of the uncertainties based on correlation analysis. This two-part analysis enables an informed decision on the suitability of RL for a control problem without training an RL agent. We apply our test to several benchmarks, demonstrating its applicability to a wide range of control problems and highlight the potential to save computational resources.