OCMDP: Observation-Constrained Markov Decision Process

TL;DR

This paper introduces OCMDP, a new framework for balancing observation costs and control in decision-making, using a model-free deep reinforcement learning approach that separates sensing and control actions.

Contribution

The paper proposes OCMDP, a novel model that jointly learns observation and control strategies with an iterative RL algorithm, reducing observation costs without environment dynamics knowledge.

Findings

Significant reduction in observation costs in simulated diagnostic tasks

Outperforms baseline methods in efficiency and cost savings

Effective in healthcare environment scenarios

Abstract

In many practical applications, decision-making processes must balance the costs of acquiring information with the benefits it provides. Traditional control systems often assume full observability, an unrealistic assumption when observations are expensive. We tackle the challenge of simultaneously learning observation and control strategies in such cost-sensitive environments by introducing the Observation-Constrained Markov Decision Process (OCMDP), where the policy influences the observability of the true state. To manage the complexity arising from the combined observation and control actions, we develop an iterative, model-free deep reinforcement learning algorithm that separates the sensing and control components of the policy. This decomposition enables efficient learning in the expanded action space by focusing on when and what to observe, as well as determining optimal control actions, without requiring knowledge of the environment's dynamics. We validate our approach on a simulated diagnostic task and a realistic healthcare environment using HeartPole. Given both scenarios, the experimental results demonstrate that our model achieves a substantial reduction in observation costs on average, significantly outperforming baseline methods by a notable margin in efficiency.