Latency-Aware Contextual Bandit: Application to Cryo-EM Data Collection

TL;DR

This paper presents a latency-aware contextual bandit framework tailored for applications like cryo-EM data collection, optimizing decision-making under delays to maximize cumulative reward.

Contribution

It introduces a novel framework and the COAF algorithm that incorporate action delays into contextual bandit problems, with theoretical regret bounds and practical validation.

Findings

The COAF algorithm effectively balances exploration and exploitation considering delays.

The approach achieves regret bounds comparable to standard contextual bandits.

Numerical experiments show improved reward maximization in cryo-EM data collection.

Abstract

We introduce a latency-aware contextual bandit framework that generalizes the standard contextual bandit problem, where the learner adaptively selects arms and switches decision sets under action delays. In this setting, the learner observes the context and may select multiple arms from a decision set, with the total time determined by the selected subset. The problem can be framed as a special case of semi-Markov decision processes (SMDPs), where contexts and latencies are drawn from an unknown distribution. Leveraging the Bellman optimality equation, we design the contextual online arm filtering (COAF) algorithm, which balances exploration, exploitation, and action latency to minimize regret relative to the optimal average-reward policy. We analyze the algorithm and show that its regret upper bounds match established results in the contextual bandit literature. In numerical experiments on a movie recommendation dataset and cryogenic electron microscopy (cryo-EM) data, we demonstrate that our approach efficiently maximizes cumulative reward over time.