Neural Value Iteration

TL;DR

This paper introduces Neural Value Iteration, a novel approach that uses neural networks to represent POMDP value functions, enabling efficient planning in large-scale problems where traditional methods are computationally infeasible.

Contribution

It proposes representing POMDP value functions as neural networks, combining neural generalization with value iteration to solve large-scale problems.

Findings

Achieves near-optimal solutions in extremely large POMDPs.

Outperforms traditional offline solvers in scalability.

Enables planning in previously intractable problems.

Abstract

The value function of a POMDP exhibits the piecewise-linear-convex (PWLC) property and can be represented as a finite set of hyperplanes, known as $\alpha$-vectors. Most state-of-the-art POMDP solvers (offline planners) follow the point-based value iteration scheme, which performs Bellman backups on $\alpha$-vectors at reachable belief points until convergence. However, since each $\alpha$-vector is $|S|$-dimensional, these methods quickly become intractable for large-scale problems due to the prohibitive computational cost of Bellman backups. In this work, we demonstrate that the PWLC property allows a POMDP's value function to be alternatively represented as a finite set of neural networks. This insight enables a novel POMDP planning algorithm called \emph{Neural Value Iteration}, which combines the generalization capability of neural networks with the classical value iteration framework. Our approach achieves near-optimal solutions even in extremely large POMDPs that are intractable for existing offline solvers.