Quadratic Truncated Random Return in Distributional LQR: Positive Definiteness, Density, and Log-Concavity

TL;DR

This paper investigates the properties of the truncated random return in distributional LQR, providing conditions for positive definiteness, eigenvalue bounds, and distributional characteristics such as log-concavity.

Contribution

It offers a quadratic form expression for the truncated random return and establishes new theoretical results on its positive definiteness and distributional properties.

Findings

Quadratic form representation of truncated random return

Conditions for positive definiteness of the associated matrix

Log-concavity of the distribution under Gaussian disturbances

Abstract

Distributional linear quadratic regulator (LQR) is a new framework that integrates the distributional reinforcement learning and classical LQR, which offers a new way to study the random return instead of the expected cost. Unlike iterative approximation using dynamic programming in the DRL, a closed-form expression for the random return can be exactly characterized in the distributional LQR, which is defined over infinitely many random variables. In recent work [1, 2], it has been shown that this random return can be well approximated by a finite number of random variables, which we call truncated random return. In this paper, we study the truncated random return in the distributional LQR. We show that the truncated random return can be naturally expressed in the quadratic form. We develop a sufficient condition for the positive definiteness of the block symmetric matrix in the quadratic form and provide the lower and upper bounds on the eigenvalues of this matrix. We further show that in this case, the truncated random return follows a positively weighted non-central chi-square distribution if the random disturbances admits Gaussian, and its cumulative distribution function is log-concave if the probability density function of the random disturbances is log-concave.