Whittle Index Policy for Crawling Ephemeral Content

TL;DR

This paper develops a Whittle index-based policy for efficiently scheduling crawlers to retrieve ephemeral online content, optimizing for user engagement metrics like clicks and search requests.

Contribution

It introduces a novel Whittle index approach for crawling ephemeral content, enabling simple and scalable scheduling policies with theoretical justification.

Findings

Derived the Whittle index for ephemeral content crawling

Provided theoretical proof of the index's validity

Demonstrated the policy's efficiency in simulations

Abstract

We consider a task of scheduling a crawler to retrieve content from several sites with ephemeral content. A user typically loses interest in ephemeral content, like news or posts at social network groups, after several days or hours. Thus, development of timely crawling policy for such ephemeral information sources is very important. We first formulate this problem as an optimal control problem with average reward. The reward can be measured in the number of clicks or relevant search requests. The problem in its initial formulation suffers from the curse of dimensionality and quickly becomes intractable even with moderate number of information sources. Fortunately, this problem admits a Whittle index, which leads to problem decomposition and to a very simple and efficient crawling policy. We derive the Whittle index and provide its theoretical justification.