Thompson Sampling-Based Learning and Control for Unknown Dynamic Systems

TL;DR

This paper introduces a novel Thompson sampling-based method for learning and controlling unknown nonlinear systems using reproducing kernel Hilbert spaces, providing theoretical guarantees and demonstrating effectiveness through numerical experiments.

Contribution

It develops a kernel-based control law learning framework with a Thompson sampling approach, extending applicability to general function spaces in control system design.

Findings

The method achieves exponential convergence in learning control laws.

The approach provides bounds on control regret.

Numerical experiments validate the method's effectiveness.

Abstract

Thompson sampling (TS) is a Bayesian randomized exploration strategy that samples options (e.g., system parameters or control laws) from the current posterior and then applies the selected option that is optimal for a task, thereby balancing exploration and exploitation; this makes TS effective for active learning-based controller design. However, TS relies on finite parametric representations, which limits its applicability to more general spaces, which are more commonly encountered in control system design. To address this issue, this work proposes a parameterization method for control law learning using reproducing kernel Hilbert spaces and designs a data-driven active learning control approach. Specifically, the proposed method treats the control law as an element in a function space, allowing the design of control laws without imposing restrictions on the system structure or the form of the controller. A TS framework is proposed in this work to reduce control costs through online exploration and exploitation, and the convergence guarantees are further provided for the learning process. Theoretical analysis shows that the proposed method learns the relationship between control laws and closed-loop performance metrics at an exponential rate, and the upper bound of control regret is also derived. Furthermore, the closed-loop stability of the proposed learning framework is analyzed. Numerical experiments on controlling unknown nonlinear systems validate the effectiveness of the proposed method.