On the function approximation error for risk-sensitive reinforcement learning

TL;DR

This paper derives new error bounds for risk-sensitive policy evaluation in reinforcement learning by leveraging Markov chain irreducibility and Perron-Frobenius theory, improving upon previous spectral bounds.

Contribution

It introduces novel bounds based on irreducibility and Perron-Frobenius eigenvectors, offering tighter error estimates than prior spectral variation bounds.

Findings

Bounds are tight and match actual errors in examples

New bounds outperform previous spectral bounds in large state spaces

Provides eigenvalue comparison bounds for irreducible matrices

Abstract

In this paper we obtain several informative error bounds on function approximation for the policy evaluation algorithm proposed by Basu et al. when the aim is to find the risk-sensitive cost represented using exponential utility. The main idea is to use classical Bapat's inequality and to use Perron-Frobenius eigenvectors (exists if we assume irreducible Markov chain) to get the new bounds. The novelty of our approach is that we use the irreduciblity of Markov chain to get the new bounds whereas the earlier work by Basu et al. used spectral variation bound which is true for any matrix. We also give examples where all our bounds achieve the "actual error" whereas the earlier bound given by Basu et al. is much weaker in comparison. We show that this happens due to the absence of difference term in the earlier bound which is always present in all our bounds when the state space is large. Additionally, we discuss how all our bounds compare with each other. As a corollary of our main result we provide a bound between largest eigenvalues of two irreducibile matrices in terms of the matrix entries.