Goal-oriented Estimation of Multiple Markov Sources in Resource-constrained Systems

TL;DR

This paper develops goal-oriented sampling policies for remote estimation of multiple Markov sources in resource-limited networks, minimizing actuation error costs through novel reinforcement learning and optimization techniques.

Contribution

It introduces a Lyapunov drift-based framework and deep reinforcement learning methods for efficient, resource-aware source updating in unreliable communication environments.

Findings

Proposed low-complexity DPP policy for known statistics.

Developed LO-DRL policy for unknown source/channel conditions.

Achieved significant reduction in uninformative transmissions.

Abstract

This paper investigates goal-oriented communication for remote estimation of multiple Markov sources in resource-constrained networks. An agent decides the updating times of the sources and transmits the packet to a remote destination over an unreliable channel with delay. The destination is tasked with source reconstruction for actuation. We utilize the metric \textit{cost of actuation error} (CAE) to capture the state-dependent actuation costs. We aim for a sampling policy that minimizes the long-term average CAE subject to an average resource constraint. We formulate this problem as an average-cost constrained Markov Decision Process (CMDP) and relax it into an unconstrained problem by utilizing \textit{Lyapunov drift} techniques. Then, we propose a low-complexity \textit{drift-plus-penalty} (DPP) policy for systems with known source/channel statistics and a Lyapunov optimization-based deep reinforcement learning (LO-DRL) policy for unknown environments. Our policies significantly reduce the number of uninformative transmissions by exploiting the timing of the important information.