# Intermittently Observable Markov Decision Processes

## TL;DR

This paper addresses decision-making in Markov Decision Processes with unreliable, intermittent state information, proposing new formulations and algorithms to find near-optimal policies efficiently despite information losses.

## Contribution

It introduces a belief MDP and a tree MDP formulation for intermittent observations, along with finite-state approximations and a nested value iteration algorithm for improved efficiency.

## Key findings

- Finite-state approximations enable near-optimal policy computation.
- Nested value iteration outperforms standard methods in speed.
- Numerical results confirm the effectiveness of proposed algorithms.

## Abstract

This paper investigates MDPs with intermittent state information. We consider a scenario where the controller perceives the state information of the process via an unreliable communication channel. The transmissions of state information over the whole time horizon are modeled as a Bernoulli lossy process. Hence, the problem is finding an optimal policy for selecting actions in the presence of state information losses. We first formulate the problem as a belief MDP to establish structural results. The effect of state information losses on the expected total discounted reward is studied systematically. Then, we reformulate the problem as a tree MDP whose state space is organized in a tree structure. Two finite-state approximations to the tree MDP are developed to find near-optimal policies efficiently. Finally, we put forth a nested value iteration algorithm for the finite-state approximations, which is proved to be faster than standard value iteration. Numerical results demonstrate the effectiveness of our methods.