A Sharp Memory-Regret Trade-Off for Multi-Pass Streaming Bandits

TL;DR

This paper investigates the trade-off between memory and regret in multi-pass streaming bandit algorithms, revealing a sharp transition where limited memory suffices for near-optimal regret, but additional memory yields minimal gains.

Contribution

It establishes tight upper and lower bounds on regret for any number of passes, uncovering a surprising phase transition in memory efficiency for streaming bandits.

Findings

O(1) memory achieves near-optimal regret with multiple passes

Increasing memory beyond o(K) offers negligible regret reduction

Lower bounds are proved using information-theoretic and round elimination techniques

Abstract

The stochastic $K$-armed bandit problem has been studied extensively due to its applications in various domains ranging from online advertising to clinical trials. In practice however, the number of arms can be very large resulting in large memory requirements for simultaneously processing them. In this paper we consider a streaming setting where the arms are presented in a stream and the algorithm uses limited memory to process these arms. Here, the goal is not only to minimize regret, but also to do so in minimal memory. Previous algorithms for this problem operate in one of the two settings: they either use $\Omega(\log \log T)$ passes over the stream (Rathod, 2021; Chaudhuri and Kalyanakrishnan, 2020; Liau et al., 2018), or just a single pass (Maiti et al., 2021). In this paper we study the trade-off between memory and regret when $B$ passes over the stream are allowed, for any $B \geq 1$, and establish tight regret upper and lower bounds for any $B$-pass algorithm. Our results uncover a surprising *sharp transition phenomenon*: $O(1)$ memory is sufficient to achieve $\widetilde\Theta\Big(T^{\frac{1}{2} + \frac{1}{2^{B+2}-2}}\Big)$ regret in $B$ passes, and increasing the memory to any quantity that is $o(K)$ has almost no impact on further reducing this regret, unless we use $\Omega(K)$ memory. Our main technical contribution is our lower bound which requires the use of information-theoretic techniques as well as ideas from round elimination to show that the *residual problem* remains challenging over subsequent passes.