Scaling Internal-State Policy-Gradient Methods for POMDPs

TL;DR

This paper introduces improved policy-gradient algorithms for learning memory-based policies in partially observable environments, demonstrating their effectiveness on large POMDPs like robot navigation and multi-agent tasks.

Contribution

It develops novel algorithms for memory-based policy learning in POMDPs, applicable both with known models and via simulation, advancing the capabilities of policy-gradient methods in complex environments.

Findings

Algorithms outperform previous methods on large POMDPs

Effective in noisy robot navigation scenarios

Applicable to multi-agent problems

Abstract

Policy-gradient methods have received increased attention recently as a mechanism for learning to act in partially observable environments. They have shown promise for problems admitting memoryless policies but have been less successful when memory is required. In this paper we develop several improved algorithms for learning policies with memory in an infinite-horizon setting -- directly when a known model of the environment is available, and via simulation otherwise. We compare these algorithms on some large POMDPs, including noisy robot navigation and multi-agent problems.