Model-free Control of Chaos with Continuous Deep Q-learning

TL;DR

This paper introduces a model-free deep reinforcement learning approach to control chaotic systems, avoiding the need for precise mathematical models and improving learning efficiency by focusing exploration within a relevant region.

Contribution

It proposes a two-step data-based control policy that efficiently stabilizes chaotic systems without requiring a mathematical model.

Findings

The method effectively stabilizes chaotic systems.

It reduces exploration time by focusing on a specific region.

The approach outperforms traditional model-based methods.

Abstract

The OGY method is one of control methods for a chaotic system. In the method, we have to calculate a stabilizing periodic orbit embedded in its chaotic attractor. Thus, we cannot use this method in the case where a precise mathematical model of the chaotic system cannot be identified. In this case, the delayed feedback control proposed by Pyragas is useful. However, even in the delayed feedback control, we need the mathematical model to determine a feedback gain that stabilizes the periodic orbit. To overcome this problem, we propose a model-free reinforcement learning algorithm to the design of a controller for the chaotic system. In recent years, model-free reinforcement learning algorithms with deep neural networks have been paid much attention to. Those algorithms make it possible to control complex systems. However, it is known that model-free reinforcement learning algorithms are not efficient because learners must explore their control policies over the entire state space. Moreover, model-free reinforcement learning algorithms with deep neural networks have the disadvantage in taking much time to learn their control optimal policies. Thus, we propose a data-based control policy consisting of two steps, where we determine a region including the stabilizing periodic orbit first, and make the controller learn an optimal control policy for its stabilization. In the proposed method, the controller efficiently explores its control policy only in the region.