Posterior Concentration Rates for Bayesian Penalized Splines

TL;DR

This paper investigates the asymptotic behavior of Bayesian penalized splines in Gaussian nonparametric regression, establishing near optimal posterior concentration rates and demonstrating the adaptivity of hyperpriors through theoretical and empirical analysis.

Contribution

It provides the first theoretical analysis of posterior concentration rates for Bayesian penalized splines with hyperpriors, introducing new concepts and methods for the analysis.

Findings

Posterior concentration can be near optimal with appropriate hyperpriors.

A Weibull hyperprior with shape 1/2 balances smoothing effectively.

Empirical results confirm the adaptivity of the hyperprior in practice.

Abstract

Despite their widespread use in practice, the asymptotic properties of Bayesian penalized splines have not been investigated so far. We close this gap and study posterior concentration rates for Bayesian penalized splines in a Gaussian nonparametric regression model. A key feature of the approach is the hyperprior on the smoothing variance, which allows for adaptive smoothing in practice but complicates the theoretical analysis considerably as it destroys conjugacy and precludes analytic expressions for the posterior moments. To derive our theoretical results, we rely on several new concepts including a carefully defined proper version of the partially improper penalized splines prior as well as an innovative spline estimator that projects the observations onto the first basis functions of a Demmler-Reinsch basis. Our results show that posterior concentration at near optimal rate can be achieved if the hyperprior on the smoothing variance strikes a fine balance between oversmoothing and undersmoothing, which can for instance be met by a Weibull hyperprior with shape parameter 1/2. We complement our theoretical results with empirical evidence demonstrating the adaptivity of the hyperprior in practice.