Recurrent Feedback Improves Feedforward Representations in Deep Neural Networks

TL;DR

Introducing recurrent feedback and horizontal connections into deep neural networks enhances their robustness to noise and occlusion, improves discriminability of object representations, and integrates semantic context into early visual processing.

Contribution

This study demonstrates that adding feedback loops to a deep CNN improves its robustness and discriminability by incorporating semantic context into early representations.

Findings

Recurrent feedback increases robustness against noise and occlusion.

Feedback improves discriminability of object classes in early network layers.

Semantic feedback enhances the network's ability to map noisy inputs to meaningful representations.

Abstract

The abundant recurrent horizontal and feedback connections in the primate visual cortex are thought to play an important role in bringing global and semantic contextual information to early visual areas during perceptual inference, helping to resolve local ambiguity and fill in missing details. In this study, we find that introducing feedback loops and horizontal recurrent connections to a deep convolution neural network (VGG16) allows the network to become more robust against noise and occlusion during inference, even in the initial feedforward pass. This suggests that recurrent feedback and contextual modulation transform the feedforward representations of the network in a meaningful and interesting way. We study the population codes of neurons in the network, before and after learning with feedback, and find that learning with feedback yielded an increase in discriminability (measured by d-prime) between the different object classes in the population codes of the neurons in the feedforward path, even at the earliest layer that receives feedback. We find that recurrent feedback, by injecting top-down semantic meaning to the population activities, helps the network learn better feedforward paths to robustly map noisy image patches to the latent representations corresponding to important visual concepts of each object class, resulting in greater robustness of the network against noises and occlusion as well as better fine-grained recognition.