Contextual Bandits for Resource-Constrained Devices using Probabilistic Learning

TL;DR

This paper introduces probabilistic HD-CB, a low-precision, resource-efficient contextual bandit algorithm suitable for deployment on constrained devices, improving scalability and decision quality.

Contribution

It proposes a probabilistic update rule for hyperdimensional CB, enabling low-precision operation without overflow and maintaining high decision accuracy.

Findings

Probabilistic HD-CB outperforms binarized HD-CB at equal precision.

It approaches the performance of full HD-CB with only 3 bits per component.

The method reduces update costs proportionally to the fraction of components updated.

Abstract

Contextual bandits (CB) are online sequential decision-making problems under partial feedback that underpin many adaptive services. There is a growing demand to deploy CB agents directly on-device, under strict constraints on memory, compute, and energy. However, standard linear CB algorithms are often impractical for resource-constrained devices with their unfavorable scaling in computational and memory costs. Recently, HD-CB, a CB approach based on hyperdimensional computing principles, has been proposed to model and solve CB problems by moving into high-dimensional spaces. HD-CB offers faster convergence, favorable scalability, and improves memory efficiency compared to linear CB algorithms. However, its learning rule is accumulation-based: the values of action vectors grow over time, requiring high precision. While periodic binarization can prevent overflow in low-precision components, it may discard important information about magnitudes and degrade decision quality. This paper introduces probabilistic HD-CB, a low-precision variant that replaces deterministic accumulation with a probabilistic update rule. At each step, only a random subset of vector components is updated, with a time-decaying update probability, and component values are constrained to a predefined range [-k,+k]. This approach enables low-precision components, prevents overflow without periodic binarization, and reduces the expected update cost in proportion to the fraction of updated components. Off-policy evaluation on standardized synthetic CB benchmarks using the Open Bandit Pipeline shows that probabilistic HD-CB consistently outperforms binarized HD-CB at equal precision, while approaching the performance of HD-CB with as few as 3 bits per component.