Multiply Accelerated Value Iteration for Non-Symmetric Affine Fixed Point Problems and application to Markov Decision Processes

TL;DR

This paper introduces accelerated algorithms for affine fixed point problems with non-symmetric matrices, applicable to Markov decision processes, achieving faster convergence rates through spectral analysis and higher-order methods.

Contribution

It develops a modified Nesterov gradient algorithm for non-self-adjoint matrices and introduces a higher-order accelerated method with improved convergence under spectral conditions.

Findings

The modified algorithm converges with an accelerated asymptotic rate for certain spectra.

The $d$th-order method achieves multiply accelerated rates under stricter spectral conditions.

Numerical experiments demonstrate the effectiveness of these accelerated schemes in MDP applications.

Abstract

We analyze a modified version of Nesterov accelerated gradient algorithm, which applies to affine fixed point problems with non self-adjoint matrices, such as the ones appearing in the theory of Markov decision processes with discounted or mean payoff criteria. We characterize the spectra of matrices for which this algorithm does converge with an accelerated asymptotic rate. We also introduce a $d$th-order algorithm, and show that it yields a multiply accelerated rate under more demanding conditions on the spectrum. We subsequently apply these methods to develop accelerated schemes for non-linear fixed point problems arising from Markov decision processes. This is illustrated by numerical experiments.