Feedback-Gated Rectified Linear Units

TL;DR

This paper introduces a biologically inspired feedback mechanism for rectified linear units in neural networks, demonstrating improved convergence, performance, and noise robustness on MNIST, with some benefits on CIFAR-10.

Contribution

The paper proposes a novel feedback gating mechanism for rectified linear units inspired by the human brain, enhancing neural network training and robustness.

Findings

Faster convergence on MNIST with feedback

Improved performance and noise robustness on MNIST

Some benefits observed on CIFAR-10

Abstract

Feedback connections play a prominent role in the human brain but have not received much attention in artificial neural network research. Here, a biologically inspired feedback mechanism which gates rectified linear units is proposed. On the MNIST dataset, autoencoders with feedback show faster convergence, better performance, and more robustness to noise compared to their counterparts without feedback. Some benefits, although less pronounced and less consistent, can be observed when networks with feedback are applied on the CIFAR-10 dataset.