The Effects of Multi-Task Learning on ReLU Neural Network Functions

TL;DR

This paper reveals that multi-task shallow ReLU neural networks often produce solutions akin to kernel regression, showing unique solutions in certain cases and connecting neural network solutions to kernel methods and Sobolev spaces.

Contribution

It establishes a novel connection between multi-task neural networks and kernel methods, proving solution uniqueness and characterizing the solutions as minimum-norm problems in Sobolev spaces.

Findings

Solutions resemble kernel regression for each task

Multi-task solutions are almost always unique

Large number of tasks lead to Hilbert space minimization

Abstract

This paper studies the properties of solutions to multi-task shallow ReLU neural network learning problems, wherein the network is trained to fit a dataset with minimal sum of squared weights. Remarkably, the solutions learned for each individual task resemble those obtained by solving a kernel regression problem, revealing a novel connection between neural networks and kernel methods. It is known that single-task neural network learning problems are equivalent to a minimum norm interpolation problem in a non-Hilbertian Banach space, and that the solutions of such problems are generally non-unique. In contrast, we prove that the solutions to univariate-input, multi-task neural network interpolation problems are almost always unique, and coincide with the solution to a minimum-norm interpolation problem in a Sobolev (Reproducing Kernel) Hilbert Space. We also demonstrate a similar phenomenon in the multivariate-input case; specifically, we show that neural network learning problems with large numbers of tasks are approximately equivalent to an $\ell^2$ (Hilbert space) minimization problem over a fixed kernel determined by the optimal neurons.