Which Sensor to Observe? Timely Tracking of a Joint Markov Source with Model Predictive Control

TL;DR

This paper addresses the challenge of optimally selecting sensors for remote tracking of a joint Markov process to minimize the age of incorrect information, using belief-based MDP and MPC methods.

Contribution

It introduces a belief-based MDP framework for sensor selection in joint Markov process tracking and proposes two MPC approaches, including a reinforcement learning method.

Findings

Proposed belief MDP for partial observability.

Developed two MPC algorithms for sensor scheduling.

Demonstrated effectiveness in minimizing MAoII.

Abstract

In this paper, we investigate the problem of remote estimation of a discrete-time joint Markov process using multiple sensors. Each sensor observes a different component of the joint Markov process, and in each time slot, the monitor obtains a partial state value by sending a pull request to one of the sensors. The monitor chooses the sequence of sensors to observe with the goal of minimizing the mean of age of incorrect information (MAoII) by using the partial state observations obtained, which have different freshness levels. For instance, a monitor may be interested in tracking the location of an object by obtaining observations from two sensors, which observe the $x$ and $y$ coordinates of the object separately, in different time slots. The monitor, then, needs to decide which coordinate to observe in the next time slot given the history. In addition to this partial observability of the state of Markov process, there is an erasure channel with a fixed one-slot delay between each sensor and the monitor. First, we obtain a sufficient statistic, namely the \emph{belief}, representing the joint distribution of the age of incorrect information (AoII) and the current state of the observed process by using the history of all pull requests and observations. Then, we formulate the problem with a continuous state-space Markov decision problem (MDP), namely belief MDP. To solve the problem, we propose two model predictive control (MPC) methods, namely MPC without terminal costs (MPC-WTC) and reinforcement learning MPC (RL-MPC), that have different advantages in implementation.