Monitoring State Transitions in Markovian Systems with Sampling Cost

TL;DR

This paper studies optimal and near-optimal strategies for monitoring Markovian systems with costly queries, proposing a greedy policy and a learning variant that balance prediction accuracy and query costs.

Contribution

It introduces a threshold-based greedy policy for state monitoring, analyzes its suboptimality, and proposes a PSGD-based learning method for unknown transition probabilities.

Findings

Greedy policy is generally suboptimal with unbounded competitive ratio.

Under common conditions, greedy performs close to optimal.

PSGD-based learning improves efficiency and performance in unknown settings.

Abstract

We consider a node-monitor pair, where the node's state varies with time. The monitor needs to track the node's state at all times; however, there is a fixed cost for each state query. So the monitor may instead predict the state using time-series forecasting methods, including time-series foundation models (TSFMs), and query only when prediction uncertainty is high. Since query decisions influence prediction accuracy, determining when to query is nontrivial. A natural approach is a greedy policy that predicts when the expected prediction loss is below the query cost and queries otherwise. We analyze this policy in a Markovian setting, where the optimal (OPT) strategy is a state-dependent threshold policy minimizing the time-averaged sum of query cost and prediction losses. We show that, in general, the greedy policy is suboptimal and can have an unbounded competitive ratio, but under common conditions such as identically distributed transition probabilities, it performs close to OPT. For the case of unknown transition probabilities, we further propose a projected stochastic gradient descent (PSGD)-based learning variant of the greedy policy, which achieves a favorable predict-query tradeoff with improved computational efficiency compared to OPT.