Analysis of Value Iteration Through Absolute Probability Sequences

TL;DR

This paper introduces a novel analysis of the Value Iteration algorithm for MDPs using absolute probability sequences, focusing on its convergence in the $L^2$ norm rather than the traditional infinity norm, providing new insights into its behavior.

Contribution

It presents a new analytical framework for Value Iteration based on absolute probability sequences, expanding understanding of its convergence properties in the $L^2$ norm.

Findings

Convergence analysis of Value Iteration in the $L^2$ norm.

New insights into the algorithm's performance and behavior.

Comparison with traditional infinity norm analysis.

Abstract

Value Iteration is a widely used algorithm for solving Markov Decision Processes (MDPs). While previous studies have extensively analyzed its convergence properties, they primarily focus on convergence with respect to the infinity norm. In this work, we use absolute probability sequences to develop a new line of analysis and examine the algorithm's convergence in terms of the $L^2$ norm, offering a new perspective on its behavior and performance.