Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex

TL;DR

This paper explores the connections between Residual Networks, Recurrent Neural Networks, and the primate visual cortex, proposing generalized architectures that are both biologically plausible and effective on image classification tasks.

Contribution

It introduces a unified framework linking ResNets and RNNs, and suggests that certain RNNs can model the ventral stream in visual cortex, with empirical validation on CIFAR-10 and ImageNet.

Findings

A shallow RNN is equivalent to a deep ResNet with weight sharing.

Shared-weight RNNs perform comparably to ResNets with fewer parameters.

Proposed architectures are effective on standard image datasets.

Abstract

We discuss relations between Residual Networks (ResNet), Recurrent Neural Networks (RNNs) and the primate visual cortex. We begin with the observation that a special type of shallow RNN is exactly equivalent to a very deep ResNet with weight sharing among the layers. A direct implementation of such a RNN, although having orders of magnitude fewer parameters, leads to a performance similar to the corresponding ResNet. We propose 1) a generalization of both RNN and ResNet architectures and 2) the conjecture that a class of moderately deep RNNs is a biologically-plausible model of the ventral stream in visual cortex. We demonstrate the effectiveness of the architectures by testing them on the CIFAR-10 and ImageNet dataset.