Towards Using Fully Observable Policies for POMDPs

TL;DR

This paper introduces a novel approach for solving POMDPs by leveraging policies from fully observable versions, using a mixture value function to handle multimodal beliefs, demonstrated on a new benchmark.

Contribution

It proposes a new mixture value function and greedy policy for POMDPs based on fully observable policies, addressing multimodal belief challenges.

Findings

Policy outperforms mode-ignoring approaches on the benchmark.

Mathematical framework for mixture value functions developed.

Benchmark built on Reconnaissance Blind TicTacToe.

Abstract

Partially Observable Markov Decision Process (POMDP) is a framework applicable to many real world problems. In this work, we propose an approach to solve POMDPs with multimodal belief by relying on a policy that solves the fully observable version. By defininig a new, mixture value function based on the value function from the fully observable variant, we can use the corresponding greedy policy to solve the POMDP itself. We develop the mathematical framework necessary for discussion, and introduce a benchmark built on the task of Reconnaissance Blind TicTacToe. On this benchmark, we show that our policy outperforms policies ignoring the existence of multiple modes.