Ergodic-risk Criterion for Stochastically Stabilizing Policy Optimization

TL;DR

This paper develops ergodic-risk criteria for long-term risk assessment in controlled Markov chains, extending risk-sensitive policy optimization to heavy-tailed noise scenarios with theoretical guarantees.

Contribution

It introduces a novel ergodic-risk framework using functional CLTs, enabling risk-sensitive policy optimization in complex stochastic systems with heavy-tailed noise.

Findings

Convergence of ergodic-risk criteria under uniform ergodicity.

Extension of risk-sensitive control to heavy-tailed noise.

Primal-dual algorithm for risk-constrained policy optimization.

Abstract

This paper introduces ergodic-risk criteria, which capture long-term cumulative risks associated with controlled Markov chains through probabilistic limit theorems--in contrast to existing methods that require assumptions of either finite hitting time, finite state/action space, or exponentiation necessitating light-tailed distributions. Using tailored Functional Central Limit Theorems (FCLT), we demonstrate that the time-correlated terms in the ergodic-risk criteria converge under uniform ergodicity and establish conditions for the convergence of these criteria in non-stationary general-state Markov chains involving heavy-tailed distributions. For quadratic risk functionals on stochastic linear systems, in addition to internal stability, this requires the (possibly heavy-tailed) process noise to have only a finite fourth moment. After quantifying cumulative uncertainties in risk functionals that account for extreme deviations, these ergodic-risk criteria are then incorporated into policy optimizations, thereby extending the standard average optimal synthesis to a risk-sensitive framework. Finally, by establishing the strong duality of the constrained policy optimization, we propose a primal-dual algorithm that optimizes average performance while ensuring that certain risks associated with these ergodic-risk criteria are constrained. Our risk-sensitive framework offers a theoretically guaranteed policy iteration for the long-term risk-sensitive control of processes involving heavy-tailed noise, which is shown to be effective through several simulations.