Hierarchical Upper Confidence Bounds for Constrained Online Learning

TL;DR

This paper introduces a hierarchical bandit framework with constraints, proposing an algorithm that achieves near-optimal regret bounds and high-probability constraint satisfaction in complex decision hierarchies.

Contribution

It extends contextual bandits to hierarchical, constrained settings and provides a novel algorithm with theoretical guarantees and minimax regret bounds.

Findings

Proposed HC-UCB algorithm for hierarchical constrained bandits.

Established sublinear regret bounds for HC-UCB.

Proved high-probability constraint satisfaction and near-optimal regret.

Abstract

The multi-armed bandit (MAB) problem is a foundational framework in sequential decision-making under uncertainty, extensively studied for its applications in areas such as clinical trials, online advertising, and resource allocation. Traditional MAB formulations, however, do not adequately capture scenarios where decisions are structured hierarchically, involve multi-level constraints, or feature context-dependent action spaces. In this paper, we introduce the hierarchical constrained bandits (HCB) framework, which extends the contextual bandit problem to incorporate hierarchical decision structures and multi-level constraints. We propose the hierarchical constrained upper confidence bound (HC-UCB) algorithm, designed to address the complexities of the HCB problem by leveraging confidence bounds within a hierarchical setting. Our theoretical analysis establishes sublinear regret bounds for HC-UCB and provides high-probability guarantees for constraint satisfaction at all hierarchical levels. Furthermore, we derive a minimax lower bound on the regret for the HCB problem, demonstrating the near-optimality of our algorithm. The results are significant for real-world applications where decision-making processes are inherently hierarchical and constrained, offering a robust and efficient solution that balances exploration and exploitation across multiple levels of decision-making.