Robust Active Measuring under Model Uncertainty

TL;DR

This paper introduces RAM-MDPs, an extension of MDPs that incorporates model uncertainty and active measurement, providing efficient heuristics and methods to optimize information gathering under uncertainty.

Contribution

The paper extends MDPs to RAM-MDPs to handle model uncertainty and partial observability, proposing efficient heuristics and methods to improve active measurement strategies.

Findings

Model uncertainty can lead to fewer measurements by agents.

Proposed methods effectively counteract measurement reduction due to uncertainty.

Empirical results show superior scalability and performance of the methods.

Abstract

Partial observability and uncertainty are common problems in sequential decision-making that particularly impede the use of formal models such as Markov decision processes (MDPs). However, in practice, agents may be able to employ costly sensors to measure their environment and resolve partial observability by gathering information. Moreover, imprecise transition functions can capture model uncertainty. We combine these concepts and extend MDPs to robust active-measuring MDPs (RAM-MDPs). We present an active-measure heuristic to solve RAM-MDPs efficiently and show that model uncertainty can, counterintuitively, let agents take fewer measurements. We propose a method to counteract this behavior while only incurring a bounded additional cost. We empirically compare our methods to several baselines and show their superior scalability and performance.