Reducing Parameter Space for Neural Network Training

TL;DR

This paper demonstrates that neural networks with ReLU or binary activations can be trained in a smaller, reduced parameter space without loss of generality, leading to more efficient optimization.

Contribution

It introduces a method to train neural networks in a reduced parameter space, simplifying the optimization process while maintaining equivalence to standard training.

Findings

Reduced parameter space maintains network expressiveness.

Training in reduced space improves optimization efficiency.

Numerical examples show enhanced training performance.

Abstract

For neural networks (NNs) with rectified linear unit (ReLU) or binary activation functions, we show that their training can be accomplished in a reduced parameter space. Specifically, the weights in each neuron can be trained on the unit sphere, as opposed to the entire space, and the threshold can be trained in a bounded interval, as opposed to the real line. We show that the NNs in the reduced parameter space are mathematically equivalent to the standard NNs with parameters in the whole space. The reduced parameter space shall facilitate the optimization procedure for the network training, as the search space becomes (much) smaller. We demonstrate the improved training performance using numerical examples.