Stochastic Bandits with ReLU Neural Networks

TL;DR

This paper introduces a novel approach for stochastic bandit problems using one-layer ReLU neural networks, achieving near-optimal regret bounds by exploiting the piecewise linear structure of ReLU activations.

Contribution

It presents the first regret guarantee of  ilde{O}(\u221a{T}) for ReLU neural network bandits and proposes algorithms that leverage the network's structure for efficient exploration.

Findings

Achieves  ilde{O}(\u221a{T}) regret with ReLU neural networks.

Introduces OFU-ReLU and OFU-ReLU+ algorithms with theoretical guarantees.

Transforms the problem into a linear bandit in a learned feature space.

Abstract

We study the stochastic bandit problem with ReLU neural network structure. We show that a $\tilde{O}(\sqrt{T})$ regret guarantee is achievable by considering bandits with one-layer ReLU neural networks; to the best of our knowledge, our work is the first to achieve such a guarantee. In this specific setting, we propose an OFU-ReLU algorithm that can achieve this upper bound. The algorithm first explores randomly until it reaches a linear regime, and then implements a UCB-type linear bandit algorithm to balance exploration and exploitation. Our key insight is that we can exploit the piecewise linear structure of ReLU activations and convert the problem into a linear bandit in a transformed feature space, once we learn the parameters of ReLU relatively accurately during the exploration stage. To remove dependence on model parameters, we design an OFU-ReLU+ algorithm based on a batching strategy, which can provide the same theoretical guarantee.