Scale invariant process regression: Towards Bayesian ML with minimal assumptions

TL;DR

This paper introduces a scale-invariant process regression method based on minimal invariance assumptions, providing a Bayesian approach that avoids kernel choices and improves extrapolation, with performance comparable to Gaussian processes.

Contribution

The paper derives a novel non-Gaussian stochastic process from invariance principles and presents a Bayesian regression method that requires no kernel selection and offers better extrapolation.

Findings

Equal performance to Gaussian process regression

Less arbitrary with no kernel choice

Potentially faster due to no kernel optimization

Abstract

Current methods for regularization in machine learning require quite specific model assumptions (e.g. a kernel shape) that are not derived from prior knowledge about the application, but must be imposed merely to make the method work. We show in this paper that regularization can indeed be achieved by assuming nothing but invariance principles (w.r.t. scaling, translation, and rotation of input and output space) and the degree of differentiability of the true function. Concretely, we derive a novel (non-Gaussian) stochastic process from the above minimal assumptions, and we present a corresponding Bayesian inference method for regression. The mean posterior turns out to be a polyharmonic spline, and the posterior process is a mixture of t-processes. Compared with Gaussian process regression, the proposed method shows equal performance and has the advantages of being (i) less arbitrary (no choice of kernel) (ii) potentially faster (no kernel parameter optimization), and (iii) having better extrapolation behavior. We believe that the proposed theory has central importance for the conceptual foundations of regularization and machine learning and also has great potential to enable practical advances in ML areas beyond regression.