Beyond the Bellman Fixed Point: Geometry and Fast Policy Identification in Value Iteration

TL;DR

This paper analyzes the geometry of Q-value iteration, showing how it rapidly identifies optimal policies by entering an invariant set, with convergence rates linked to spectral properties.

Contribution

It introduces a geometric and spectral analysis of Q-VI, revealing finite-time policy identification and conditions for faster convergence than classical methods.

Findings

Q-VI reaches the optimal action class in finite time.

Distance to the invariant set decreases exponentially with a rate related to spectral radius.

Spectral and graph-theoretic conditions determine when convergence is faster than the classical rate.

Abstract

Q-value iteration (Q-VI) is usually analyzed through the \(\gamma\)-contraction of the Bellman operator. This argument proves convergence to \(Q^*\), but it gives only a coarse account of when the induced greedy policy becomes optimal. We study discounted Q-VI as a switching system and focus on the practically optimal solution set (POSS), the set of \(Q\)-functions whose tie-broken greedy policies are optimal. The main result shows that Q-VI reaches the optimal action class in finite time by entering an invariant tube around \(\mathcal X_1=Q^*+\operatorname{span}(\mathbf 1)\), which is contained in the POSS. For every \(\varepsilon>0\), the distance to \(\mathcal X_1\) satisfies an exponential bound with rate \((\bar\rho+\varepsilon)^k\), where \(\bar\rho\) is the joint spectral radius of the projected switching family restricted to directions transverse to \(\mathcal X_1\). When \(\bar\rho<\gamma\), this transverse convergence is faster than the classical contraction rate. The analysis separates fast policy identification from the subsequent convergence to \(Q^*\), which may still be governed by the all-ones mode. We also give spectral and graph-theoretic conditions under which the strict inequality \(\bar\rho<\gamma\) holds or fails.