Solving the Model Unavailable MARE using Q-Learning Algorithm

TL;DR

This paper introduces a novel iterative method combined with Q-learning to solve the modified algebraic Riccati equation without system model knowledge, validated through numerical simulation.

Contribution

A new iterative approach for solving the MARE using Q-learning that does not require system model information, applicable to both single-input and multi-input cases.

Findings

The iterative method converges to the stabilizing solution of the MARE.

Q-learning effectively solves the MARE using only input/output data.

Numerical simulations confirm the method's validity and effectiveness.

Abstract

In this paper, the discrete-time modified algebraic Riccati equation (MARE) is solved when the system model is completely unavailable. To achieve this, firstly a brand new iterative method based on the standard discrete-time algebraic Riccati equation (DARE) and its input weighting matrix is proposed to solve the MARE. For the single-input case, the iteration can be initialized by an arbitrary positive input weighting if and only if the MARE has a stabilizing solution; nevertheless a pre-given input weighting matrix of a sufficiently large magnitude is used to perform the iteration for the multi-input case when the characteristic parameter belongs to a specified subset. Benefit from the developed specific iteration structure, the Q-learning (QL) algorithm can be employed to subtly solve the MARE where only the system input/output data is used thus the system model is not required. Finally, a numerical simulation example is given to verify the effectiveness of the theoretical results and the algorithm.