Meta-Learning Hypothesis Spaces for Sequential Decision-making

TL;DR

This paper introduces Meta-KeL, a method to learn kernels from offline data for sequential decision-making, improving confidence sets and regret bounds in bandit problems without prior kernel assumptions.

Contribution

It proposes a meta-learning approach to estimate kernels from offline data, ensuring valid confidence sets and competitive regret bounds in Bayesian optimization.

Findings

Meta-KeL produces confidence sets as tight as those with true kernels.

The method achieves regret bounds comparable to known kernel settings.

Empirical results validate the effectiveness of the learned kernels.

Abstract

Obtaining reliable, adaptive confidence sets for prediction functions (hypotheses) is a central challenge in sequential decision-making tasks, such as bandits and model-based reinforcement learning. These confidence sets typically rely on prior assumptions on the hypothesis space, e.g., the known kernel of a Reproducing Kernel Hilbert Space (RKHS). Hand-designing such kernels is error prone, and misspecification may lead to poor or unsafe performance. In this work, we propose to meta-learn a kernel from offline data (Meta-KeL). For the case where the unknown kernel is a combination of known base kernels, we develop an estimator based on structured sparsity. Under mild conditions, we guarantee that our estimated RKHS yields valid confidence sets that, with increasing amounts of offline data, become as tight as those given the true unknown kernel. We demonstrate our approach on the kernelized bandit problem (a.k.a.~Bayesian optimization), where we establish regret bounds competitive with those given the true kernel. We also empirically evaluate the effectiveness of our approach on a Bayesian optimization task.