Half-Layered Neural Networks

TL;DR

This paper introduces a novel neural network layer called the 'half layer' that combines fixed random weights with trainable weights, demonstrating promising results on MNIST and FashionMNIST with fewer parameters.

Contribution

It proposes a new 'half' layer architecture with fixed and trainable weights, applicable in deep networks, and shows its effectiveness on standard datasets.

Findings

Achieves reasonable accuracy with fewer parameters.

Demonstrates the regularizing effect of randomized connections.

Applicable in various layers of deep networks.

Abstract

We propose a ``half'' layer of hidden units that has some of its weights randomly set and some of them trained. A half unit is composed of two stages: First, it takes a weighted sum of its inputs with fixed random weights, and second, the total activation is multiplied and then translated using two modifiable weights, before the result is passed through a nonlinearity. The number of modifiable weights of each hidden unit is thus two and does not depend on the fan-in. We show how such half units can be used in the first or any later layer in a deep network, possibly following convolutional layers. Our experiments on MNIST and FashionMNIST data sets indicate the promise of half layers, where we can achieve reasonable accuracy with a reduced number of parameters due to the regularizing effect of the randomized connections.