Using Linear Regression for Iteratively Training Neural Networks

TL;DR

This paper introduces a linear regression-based method for training neural networks, offering an alternative to backpropagation, with promising stability and speed in simple scenarios, and potential for extension to complex architectures.

Contribution

The paper proposes a novel linear regression approach for neural network training, bypassing gradient descent, and demonstrates its effectiveness on simple models with potential for broader application.

Findings

More stable than gradient-based methods in small problems

Faster convergence observed in initial experiments

Applicable to simple feedforward neural networks

Abstract

We present a simple linear regression based approach for learning the weights and biases of a neural network, as an alternative to standard gradient based backpropagation. The present work is exploratory in nature, and we restrict the description and experiments to (i) simple feedforward neural networks, (ii) scalar (single output) regression problems, and (iii) invertible activation functions. However, the approach is intended to be extensible to larger, more complex architectures. The key idea is the observation that the input to every neuron in a neural network is a linear combination of the activations of neurons in the previous layer, as well as the parameters (weights and biases) of the layer. If we are able to compute the ideal total input values to every neuron by working backwards from the output, we can formulate the learning problem as a linear least squares problem which iterates between updating the parameters and the activation values. We present an explicit algorithm that implements this idea, and we show that (at least for small problems) the approach is more stable and faster than gradient-based methods.