Optimizing Static and Adaptive Probing Schedules for Rapid Event Detection

TL;DR

This paper introduces a framework for optimizing probing schedules in large systems to rapidly detect new events, providing theoretical bounds and adaptive algorithms that improve detection efficiency.

Contribution

It formulates the Schedule Optimization problem, derives lower bounds, constructs near-optimal schedules, and develops an adaptive algorithm that learns and converges to optimal probing strategies.

Findings

Derived lower bounds for optimal schedules

Constructed near-optimal probing schedules with guarantees

Developed an adaptive algorithm that learns system parameters

Abstract

We formulate and study a fundamental search and detection problem, Schedule Optimization, motivated by a variety of real-world applications, ranging from monitoring content changes on the web, social networks, and user activities to detecting failure on large systems with many individual machines. We consider a large system consists of many nodes, where each node has its own rate of generating new events, or items. A monitoring application can probe a small number of nodes at each step, and our goal is to compute a probing schedule that minimizes the expected number of undiscovered items at the system, or equivalently, minimizes the expected time to discover a new item in the system. We study the Schedule Optimization problem both for deterministic and randomized memoryless algorithms. We provide lower bounds on the cost of an optimal schedule and construct close to optimal schedules with rigorous mathematical guarantees. Finally, we present an adaptive algorithm that starts with no prior information on the system and converges to the optimal memoryless algorithms by adapting to observed data.