Empirical analysis of representation learning and exploration in neural kernel bandits

TL;DR

This paper investigates the use of neural kernels in bandit algorithms, demonstrating their effectiveness in nonlinear decision tasks and providing a framework to analyze their representation learning and exploration capabilities.

Contribution

It introduces NK-based bandits that outperform existing methods, and proposes a framework to evaluate their representation learning and exploration abilities.

Findings

NK bandits achieve state-of-the-art performance on nonlinear data

The framework separates representation learning from exploration

Training frequency and model partitioning affect performance

Abstract

Neural bandits have been shown to provide an efficient solution to practical sequential decision tasks that have nonlinear reward functions. The main contributor to that success is approximate Bayesian inference, which enables neural network (NN) training with uncertainty estimates. However, Bayesian NNs often suffer from a prohibitive computational overhead or operate on a subset of parameters. Alternatively, certain classes of infinite neural networks were shown to directly correspond to Gaussian processes (GP) with neural kernels (NK). NK-GPs provide accurate uncertainty estimates and can be trained faster than most Bayesian NNs. We propose to guide common bandit policies with NK distributions and show that NK bandits achieve state-of-the-art performance on nonlinear structured data. Moreover, we propose a framework for measuring independently the ability of a bandit algorithm to learn representations and explore, and use it to analyze the impact of NK distributions w.r.t.~those two aspects. We consider policies based on a GP and a Student's t-process (TP). Furthermore, we study practical considerations, such as training frequency and model partitioning. We believe our work will help better understand the impact of utilizing NKs in applied settings.