On Provably Robust Meta-Bayesian Optimization

TL;DR

This paper introduces two robust meta-Bayesian optimization algorithms that leverage previous tasks to accelerate optimization while maintaining provable robustness against dissimilar tasks, supported by theoretical guarantees and empirical results.

Contribution

The paper proposes scalable, provably robust meta-BO algorithms, RM-GP-UCB and RM-GP-TS, with theoretical regret guarantees and adaptive weighting for enhanced robustness.

Findings

RM-GP-UCB is more robust than RM-GP-TS both theoretically and empirically.

Both algorithms perform well across various applications.

Adaptive weighting further improves robustness by diminishing dissimilar task impact.

Abstract

Bayesian optimization (BO) has become popular for sequential optimization of black-box functions. When BO is used to optimize a target function, we often have access to previous evaluations of potentially related functions. This begs the question as to whether we can leverage these previous experiences to accelerate the current BO task through meta-learning (meta-BO), while ensuring robustness against potentially harmful dissimilar tasks that could sabotage the convergence of BO. This paper introduces two scalable and provably robust meta-BO algorithms: robust meta-Gaussian process-upper confidence bound (RM-GP-UCB) and RM-GP-Thompson sampling (RM-GP-TS). We prove that both algorithms are asymptotically no-regret even when some or all previous tasks are dissimilar to the current task, and show that RM-GP-UCB enjoys a better theoretical robustness than RM-GP-TS. We also exploit the theoretical guarantees to optimize the weights assigned to individual previous tasks through regret minimization via online learning, which diminishes the impact of dissimilar tasks and hence further enhances the robustness. Empirical evaluations show that (a) RM-GP-UCB performs effectively and consistently across various applications, and (b) RM-GP-TS, despite being less robust than RM-GP-UCB both in theory and in practice, performs competitively in some scenarios with less dissimilar tasks and is more computationally efficient.