Improving Generalization with Flat Hilbert Bayesian Inference

TL;DR

This paper introduces Flat Hilbert Bayesian Inference (FHBI), a novel algorithm that improves generalization in Bayesian inference by leveraging functional perturbations in a reproducing kernel Hilbert space, supported by theoretical and empirical validation.

Contribution

The paper presents FHBI, a new Bayesian inference method that extends generalization analysis to infinite-dimensional spaces and demonstrates superior performance on diverse benchmarks.

Findings

FHBI outperforms nine baseline methods on VTAB-1K.

Theoretical analysis extends generalization bounds to functional spaces.

Empirical results show consistent improvements across datasets.

Abstract

We introduce Flat Hilbert Bayesian Inference (FHBI), an algorithm designed to enhance generalization in Bayesian inference. Our approach involves an iterative two-step procedure with an adversarial functional perturbation step and a functional descent step within a reproducing kernel Hilbert space. This methodology is supported by a theoretical analysis that extends previous findings on generalization ability from finite-dimensional Euclidean spaces to infinite-dimensional functional spaces. To evaluate the effectiveness of FHBI, we conduct comprehensive comparisons against nine baseline methods on the \texttt{VTAB-1K} benchmark, which encompasses 19 diverse datasets across various domains with diverse semantics. Empirical results demonstrate that FHBI consistently outperforms the baselines by notable margins, highlighting its practical efficacy.