# Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks

## TL;DR

This paper explains why overparameterized models like wide neural networks can generalize well using kernel regression and statistical mechanics.

## Contribution

The paper derives an analytical expression for generalization error in kernel regression using statistical mechanics.

## Key findings

- Kernel regression has an inductive bias towards simple functions.
- Generalization can worsen with more data if the data is noisy or incompatible with the kernel.
- Learning curves can have multiple peaks depending on data and kernel compatibility.

## Abstract

A theoretical understanding of generalization remains an open problem for many machine learning models, including deep networks where overparameterization leads to better performance, contradicting the conventional wisdom from classical statistics. Here, we investigate generalization error for kernel regression, which, besides being a popular machine learning method, also describes certain infinitely overparameterized neural networks. We use techniques from statistical mechanics to derive an analytical expression for generalization error applicable to any kernel and data distribution. We present applications of our theory to real and synthetic datasets, and for many kernels including those that arise from training deep networks in the infinite-width limit. We elucidate an inductive bias of kernel regression to explain data with simple functions, characterize whether a kernel is compatible with a learning task, and show that more data may impair generalization when noisy or not expressible by the kernel, leading to non-monotonic learning curves with possibly many peaks.

Canatar et al. propose a predictive theory of generalization in kernel regression applicable to real data. This theory explains various generalization phenomena observed in wide neural networks, which admit a kernel limit and generalize well despite being overparameterized.