Pointwise confidence estimation in the non-linear $\ell^2$-regularized least squares

TL;DR

This paper develops a non-asymptotic, pointwise confidence estimation method for non-linear least squares with $ ext{l}^2$ regularization, accounting for the test input's similarity to training data, and demonstrates its effectiveness empirically.

Contribution

It introduces a novel confidence bound that scales with input similarity in the feature space and provides an efficient computation method, extending classical linear confidence intervals to non-linear settings.

Findings

The confidence bound adapts to the test input's distance from training data.

Empirical results show improved coverage/width trade-off over bootstrap methods.

The method is computationally efficient, close to gradient computation cost.

Abstract

We consider a high-probability non-asymptotic confidence estimation in the $\ell^2$-regularized non-linear least-squares setting with fixed design. In particular, we study confidence estimation for local minimizers of the regularized training loss. We show a pointwise confidence bound, meaning that it holds for the prediction on any given fixed test input $x$. Importantly, the proposed confidence bound scales with similarity of the test input to the training data in the implicit feature space of the predictor (for instance, becoming very large when the test input lies far outside of the training data). This desirable last feature is captured by the weighted norm involving the inverse-Hessian matrix of the objective function, which is a generalized version of its counterpart in the linear setting, $x^{\top} \text{Cov}^{-1} x$. Our generalized result can be regarded as a non-asymptotic counterpart of the classical confidence interval based on asymptotic normality of the MLE estimator. We propose an efficient method for computing the weighted norm, which only mildly exceeds the cost of a gradient computation of the loss function. Finally, we complement our analysis with empirical evidence showing that the proposed confidence bound provides better coverage/width trade-off compared to a confidence estimation by bootstrapping, which is a gold-standard method in many applications involving non-linear predictors such as neural networks.