Kernel-based guarantees for nonlinear parametric models in Bayesian optimization

TL;DR

This paper introduces a kernel-based framework to provide theoretical guarantees for nonlinear parametric models in Bayesian optimization, addressing a gap in existing analyses.

Contribution

It develops a unified kernel-based approach to analyze regularized nonlinear models trained on adaptively collected data, supporting convergence guarantees.

Findings

Confidence bounds for nonlinear models with convex losses

Supports convergence guarantees for nonlinear acquisition policies

Unifies analysis of nonlinear models in Bayesian optimization

Abstract

Modern Bayesian optimization and adaptive sampling methods increasingly rely on nonlinear parametric models, yet theoretical guarantees for such models under adaptive data collection remain limited. Existing analyses largely focus on Gaussian processes, kernel machines, linear models, or linearized neural approximations, leaving a gap between theory and the nonlinear models used in practice. We develop a kernel based framework for analyzing regularized nonlinear parametric models trained on adaptively collected data. Our approach uses kernels over the parameter space to induce reproducing kernel Hilbert space structures over the corresponding model class, yielding confidence bounds for models trained with broad classes of regularized convex losses. We show how these bounds can support convergence guarantees for nonlinear acquisition and surrogate models, including randomized regularized policies that select points by maximizing a trained random model. These results provide a unified route to analyzing nonlinear parametric models in Bayesian optimization and related adaptive optimization settings.