Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum

TL;DR

This paper introduces Modified Spectrum Kernels (MSKs) to adjust the inductive bias of wide neural networks by manipulating the kernel's spectrum, enabling faster training without affecting the final learned function.

Contribution

The paper proposes a novel family of kernels, MSKs, and a preconditioned gradient descent method to modify neural network bias and accelerate training.

Findings

Enables polynomial and exponential training speedups

Maintains the final solution despite bias modification

Efficient and easy to implement method

Abstract

Wide neural networks are biased towards learning certain functions, influencing both the rate of convergence of gradient descent (GD) and the functions that are reachable with GD in finite training time. As such, there is a great need for methods that can modify this bias according to the task at hand. To that end, we introduce Modified Spectrum Kernels (MSKs), a novel family of constructed kernels that can be used to approximate kernels with desired eigenvalues for which no closed form is known. We leverage the duality between wide neural networks and Neural Tangent Kernels and propose a preconditioned gradient descent method, which alters the trajectory of GD. As a result, this allows for a polynomial and, in some cases, exponential training speedup without changing the final solution. Our method is both computationally efficient and simple to implement.