Adaptive Bayesian Regression on Data with Low Intrinsic Dimensionality

TL;DR

This paper demonstrates that Gaussian process priors can adapt to the intrinsic low-dimensional structure of data, achieving optimal posterior contraction rates without explicit knowledge of the data's intrinsic dimension.

Contribution

The authors establish conditions for adaptive posterior contraction rates based on intrinsic data dimension and introduce an empirical Bayes method that bypasses explicit dimension estimation.

Findings

Achieves near-optimal adaptive contraction rates for Gaussian process priors.

Proposes an empirical Bayes approach using kernel affinity and k-nearest neighbors.

Demonstrates efficiency through numerical experiments.

Abstract

We study how the posterior contraction rate under a Gaussian process (GP) prior depends on the intrinsic dimension of the predictors and the smoothness of the regression function. An open question is whether a generic GP prior that does not incorporate knowledge of the intrinsic lower-dimensional structure of the predictors can attain an adaptive rate for a broad class of such structures. We show that this is indeed the case, establishing conditions under which the posterior contraction rates become adaptive to the intrinsic dimension in terms of the covering number of the data domain (the Minkowski dimension) and prove the nonparametric posterior contraction rate, up to a logarithmic factor. When the domain is a compact manifold, we prove the RKHS approximation to intrinsically defined H\"older functions on the manifold of any order of smoothness by a novel analysis, leading to the optimal adaptive posterior contraction rate. We propose an empirical Bayes prior on the kernel bandwidth using kernel affinity and $k$-nearest neighbor statistics, bypassing explicit estimation of the intrinsic dimension. The efficiency of the proposed Bayesian regression approach is demonstrated in various numerical experiments.