Switching-Geometry Analysis of Deflated Q-Value Iteration

TL;DR

This paper introduces a spectral radius framework for analyzing the convergence of deflated Q-value iteration in Markov decision processes, revealing sharper bounds and geometric insights.

Contribution

It provides the first JSR-based convergence analysis of deflated Q-VI, showing how deflation leads to more precise convergence rate characterization.

Findings

JSR of standard Q-VI equals the discount factor γ

Projected switching system can have JSR less than γ

Deflation yields sharper convergence bounds without changing optimal policies

Abstract

This paper develops a joint spectral radius (JSR) framework for analyzing rank-one deflated Q-value iteration (Q-VI) in discounted Markov decision process control. Focusing on an all-ones residual correction, we interpret the resulting algorithm through the geometry of switching systems and, to the best of our knowledge, give the first JSR-based convergence analysis of deflated Q-VI for policy optimization problems. Our analysis reveals that the standard Q-VI switching system model has JSR exactly the discount factor $\gamma\in (0,1)$, since all admissible subsystems share the all-ones vector as an invariant direction. By passing to the quotient space that removes this direction, we obtain a projected switching system model whose JSR governs the relevant error dynamics and may be strictly smaller than $\gamma$. Therefore, the deflated Q-VI admits a potentially sharper convergence-rate characterization than the ambient-space $\gamma$-bound. Finally, we prove that the correction is equivalent to a scalar recentering of standard Q-VI. Hence, the projected trajectory, and therefore the greedy-policy sequence, is unchanged relative to standard Q-VI initialized from the same point. The benefit of deflation is not a change in the induced decision-making problem, but a more precise JSR-based description of the convergence geometry after the redundant all-ones component is removed.