Deflated Dynamics Value Iteration

TL;DR

This paper introduces Deflated Dynamics Value Iteration (DDVI), a novel method that accelerates convergence in value iteration by removing dominant eigen-structures, and extends it to reinforcement learning with the DDTD algorithm.

Contribution

The paper proposes DDVI using matrix deflation to improve convergence rates in value iteration and extends it to RL with DDTD, demonstrating empirical effectiveness.

Findings

DDVI achieves faster convergence rates than standard VI.

Theoretical analysis shows convergence rate depends on the eigenvalues of the transition matrix.

Empirical results confirm improved performance in RL tasks.

Abstract

The Value Iteration (VI) algorithm is an iterative procedure to compute the value function of a Markov decision process, and is the basis of many reinforcement learning (RL) algorithms as well. As the error convergence rate of VI as a function of iteration $k$ is $O(\gamma^k)$, it is slow when the discount factor $\gamma$ is close to $1$. To accelerate the computation of the value function, we propose Deflated Dynamics Value Iteration (DDVI). DDVI uses matrix splitting and matrix deflation techniques to effectively remove (deflate) the top $s$ dominant eigen-structure of the transition matrix $\mathcal{P}^{\pi}$. We prove that this leads to a $\tilde{O}(\gamma^k |\lambda_{s+1}|^k)$ convergence rate, where $\lambda_{s+1}$is $(s+1)$-th largest eigenvalue of the dynamics matrix. We then extend DDVI to the RL setting and present Deflated Dynamics Temporal Difference (DDTD) algorithm. We empirically show the effectiveness of the proposed algorithms.