Universality of Gradient Descent Neural Network Training

TL;DR

This paper proves a universality result showing that any neural network's optimal weights for a classification task can be reproduced by gradient descent on an extended network, highlighting theoretical potential for meta-learning.

Contribution

It introduces a theoretical universality result demonstrating that extended neural networks can replicate any weights found by other algorithms through gradient descent.

Findings

Gradient descent can reproduce any optimal weights with network extension.

The result provides theoretical insights into meta-learning possibilities.

Design choices influence training success, but universality offers a broader perspective.

Abstract

It has been observed that design choices of neural networks are often crucial for their successful optimization. In this article, we therefore discuss the question if it is always possible to redesign a neural network so that it trains well with gradient descent. This yields the following universality result: If, for a given network, there is any algorithm that can find good network weights for a classification task, then there exists an extension of this network that reproduces these weights and the corresponding forward output by mere gradient descent training. The construction is not intended for practical computations, but it provides some orientation on the possibilities of meta-learning and related approaches.