# Bootstrap Policy Iteration for Stochastic LQ Tracking with Multiplicative Noise

## TL;DR

This paper introduces a bootstrap policy iteration algorithm for optimal tracking control of stochastic linear systems with multiplicative noise, enabling model-free reinforcement learning and data-driven computation of control gains.

## Contribution

It develops a two-phase bootstrap policy iteration method and a data-driven off-policy RL approach for stochastic LQ tracking, including a novel method for systems with state-dependent noise.

## Key findings

- Converges to the optimal feedback gain under interval excitation.
- Ensures stability and optimality in stochastic LQ tracking.
- Validated through numerical examples demonstrating effectiveness.

## Abstract

This paper studies the optimal tracking control problem for continuous-time stochastic linear systems with multiplicative noise. The solution framework involves solving a stochastic algebraic Riccati equation for the feedback gain and a Sylvester equation for the feedforward gain. To enable model-free optimal tracking, we first develop a two-phase bootstrap policy iteration (B-PI) algorithm, which bootstraps a stabilizing control gain from the trivially initialized zero-value start and proceeds with standard policy iteration. Building on this algorithm, we propose a data-driven, off-policy reinforcement learning approach that ensures convergence to the optimal feedback gain under the interval excitation condition. We further introduce a data-driven method to compute the feedforward using the obtained feedback gain. Additionally, for systems with state-dependent noise, we propose a shadow system-based optimal tracking method to eliminate the need for probing noise. The effectiveness of the proposed methods is demonstrated through numerical examples.