Track-MDP: Reinforcement Learning for Target Tracking with Controlled Sensing

TL;DR

This paper introduces Track-MDP, a novel Markov Decision Process formulation for target tracking with controlled sensing, enabling reinforcement learning solutions even when the target's motion model is unknown, and guarantees effective tracking.

Contribution

The paper proposes Track-MDP, a new MDP-based framework for target tracking that simplifies POMDP complexities and is solvable via reinforcement learning, ensuring reliable tracking performance.

Findings

Track-MDP achieves the same infinite horizon reward as the optimal POMDP.

RL-based Track-MDP policies can accurately track targets in simulations.

The method guarantees tracking of all significant target paths with certainty.

Abstract

State of the art methods for target tracking with sensor management (or controlled sensing) are model-based and are obtained through solutions to Partially Observable Markov Decision Process (POMDP) formulations. In this paper a Reinforcement Learning (RL) approach to the problem is explored for the setting where the motion model for the object/target to be tracked is unknown to the observer. It is assumed that the target dynamics are stationary in time, the state space and the observation space are discrete, and there is complete observability of the location of the target under certain (a priori unknown) sensor control actions. Then, a novel Markov Decision Process (MDP) rather than POMDP formulation is proposed for the tracking problem with controlled sensing, which is termed as Track-MDP. In contrast to the POMDP formulation, the Track-MDP formulation is amenable to an RL based solution. It is shown that the optimal policy for the Track-MDP formulation, which is approximated through RL, is guaranteed to track all significant target paths with certainty. The Track-MDP method is then compared with the optimal POMDP policy, and it is shown that the infinite horizon tracking reward of the optimal Track-MDP policy is the same as that of the optimal POMDP policy. In simulations it is demonstrated that Track-MDP based RL leads to a policy that can track the target with high accuracy.