Control Synthesis with Reinforcement Learning: A Modeling Perspective

TL;DR

This paper highlights the importance of accurate modeling in reinforcement learning-based control synthesis, showing that controllers built on precise models are more robust and reliable in real-world applications than those based on inaccurate models.

Contribution

It demonstrates the critical role of model accuracy in reinforcement learning control design and introduces empirical methods to assess robustness and sensitivity.

Findings

Controllers with accurate models are robust against disturbances.

Poor models lead to controllers that fail in physical experiments.

Sensitivity analysis helps visualize robustness regions.

Abstract

Controllers designed with reinforcement learning can be sensitive to model mismatch. We demonstrate that designing such controllers in a virtual simulation environment with an inaccurate model is not suitable for deployment in a physical setup. Controllers designed using an accurate model is robust against disturbance and small mismatch between the physical setup and the mathematical model derived from first principles; while a poor model results in a controller that performs well in simulation but fails in physical experiments. Sensitivity analysis is used to justify these discrepancies and an empirical region of attraction estimation help us visualize their robustness.