Modulation-Enhanced Excitation for Continuous-Time Reinforcement Learning via Symmetric Kronecker Products

TL;DR

This paper introduces a modulation-enhanced excitation framework using symmetric Kronecker products to improve persistent excitation in continuous-time reinforcement learning, leading to better control synthesis and stability guarantees.

Contribution

The paper develops a novel theoretical framework based on symmetric Kronecker products that systematically enhances excitation in CT-RL algorithms, improving performance and stability.

Findings

Significant improvement in conditioning for Hamilton-Jacobi-Bellman equation solutions

Enhanced control performance demonstrated on hypersonic vehicle application

Retains convergence and stability guarantees of existing EIRL algorithms

Abstract

This work introduces new results in continuous-time reinforcement learning (CT-RL) control of affine nonlinear systems to address a major algorithmic challenge due to a lack of persistence of excitation (PE). This PE design limitation has previously stifled CT-RL numerical performance and prevented these algorithms from achieving control synthesis goals. Our new theoretical developments in symmetric Kronecker products enable a proposed modulation-enhanced excitation (MEE) framework to make PE significantly more systematic and intuitive to achieve for real-world designers. MEE is applied to the suite of recently-developed excitable integral reinforcement learning (EIRL) algorithms, yielding a class of enhanced high-performance CT-RL control design methods which, due to the symmetric Kronecker product algebra, retain EIRL's convergence and closed-loop stability guarantees. Through numerical evaluation studies, we demonstrate how our new MEE framework achieves substantial improvements in conditioning when approximately solving the Hamilton-Jacobi-Bellman equation to obtain optimal controls. We use an intuitive example to provide insights on the central excitation issue under discussion, and we demonstrate the effectiveness of the proposed procedure on a real-world hypersonic vehicle (HSV) application.