Learning curves for Gaussian process regression with power-law priors and targets

TL;DR

This paper analyzes the asymptotic learning curves of Gaussian process regression with power-law spectral assumptions, providing insights applicable to infinitely wide neural networks and supporting the theory with toy experiments.

Contribution

It characterizes the power-law asymptotics of GPR learning curves and connects these results to neural networks via kernel methods.

Findings

Learning curves follow power-law decay under spectral assumptions.

Theoretical results apply to neural networks through kernel equivalences.

Toy experiments validate the theoretical predictions.

Abstract

We characterize the power-law asymptotics of learning curves for Gaussian process regression (GPR) under the assumption that the eigenspectrum of the prior and the eigenexpansion coefficients of the target function follow a power law. Under similar assumptions, we leverage the equivalence between GPR and kernel ridge regression (KRR) to show the generalization error of KRR. Infinitely wide neural networks can be related to GPR with respect to the neural network GP kernel and the neural tangent kernel, which in several cases is known to have a power-law spectrum. Hence our methods can be applied to study the generalization error of infinitely wide neural networks. We present toy experiments demonstrating the theory.