Efficient Learning of POMDPs with Known Observation Model in Average-Reward Setting

TL;DR

This paper introduces an efficient method for learning average-reward POMDPs with known observation models, using spectral estimation and an exploration strategy that guarantees low regret and scales well with problem size.

Contribution

It proposes the OAS spectral estimation technique and the OAS-UCRL algorithm, providing the first regret guarantees for POMDPs with known observation models in the average-reward setting.

Findings

Regret bound of order O(√T log T) for the proposed algorithm.

Efficient scaling with state, action, and observation space dimensions.

Numerical simulations validate the approach against baselines.

Abstract

Dealing with Partially Observable Markov Decision Processes is notably a challenging task. We face an average-reward infinite-horizon POMDP setting with an unknown transition model, where we assume the knowledge of the observation model. Under this assumption, we propose the Observation-Aware Spectral (OAS) estimation technique, which enables the POMDP parameters to be learned from samples collected using a belief-based policy. Then, we propose the OAS-UCRL algorithm that implicitly balances the exploration-exploitation trade-off following the $\textit{optimism in the face of uncertainty}$ principle. The algorithm runs through episodes of increasing length. For each episode, the optimal belief-based policy of the estimated POMDP interacts with the environment and collects samples that will be used in the next episode by the OAS estimation procedure to compute a new estimate of the POMDP parameters. Given the estimated model, an optimization oracle computes the new optimal policy. We show the consistency of the OAS procedure, and we prove a regret guarantee of order $\mathcal{O}(\sqrt{T \log(T)})$ for the proposed OAS-UCRL algorithm. We compare against the oracle playing the optimal stochastic belief-based policy and show the efficient scaling of our approach with respect to the dimensionality of the state, action, and observation space. We finally conduct numerical simulations to validate and compare the proposed technique with other baseline approaches.