A ReLU Dense Layer to Improve the Performance of Neural Networks

TL;DR

This paper introduces ReDense, a simple ReLU-based dense layer that enhances neural network performance by reducing training loss without increasing generalization error, applicable across various architectures and datasets.

Contribution

ReDense is a novel, low-complexity layer leveraging random weights and ReLU to improve trained neural networks' performance without vanishing gradients.

Findings

ReDense reduces training loss across multiple architectures.

ReDense improves test performance on benchmark datasets.

ReDense maintains small generalization error.

Abstract

We propose ReDense as a simple and low complexity way to improve the performance of trained neural networks. We use a combination of random weights and rectified linear unit (ReLU) activation function to add a ReLU dense (ReDense) layer to the trained neural network such that it can achieve a lower training loss. The lossless flow property (LFP) of ReLU is the key to achieve the lower training loss while keeping the generalization error small. ReDense does not suffer from vanishing gradient problem in the training due to having a shallow structure. We experimentally show that ReDense can improve the training and testing performance of various neural network architectures with different optimization loss and activation functions. Finally, we test ReDense on some of the state-of-the-art architectures and show the performance improvement on benchmark datasets.