Hierarchical Residuals Exploit Brain-Inspired Compositionality

TL;DR

This paper introduces Hierarchical Residual Networks (HiResNets), inspired by brain structure, which enhance deep convolutional networks with long-range residual connections, leading to improved accuracy and learning efficiency.

Contribution

The paper proposes a novel hierarchical residual architecture that mimics brain connectivity, demonstrating its effectiveness across various models and tasks.

Findings

Hierarchical residuals improve accuracy in deep networks.

Models learn hierarchical compositionality through skip connections.

Faster convergence observed with HiResNets.

Abstract

We present Hierarchical Residual Networks (HiResNets), deep convolutional neural networks with long-range residual connections between layers at different hierarchical levels. HiResNets draw inspiration on the organization of the mammalian brain by replicating the direct connections from subcortical areas to the entire cortical hierarchy. We show that the inclusion of hierarchical residuals in several architectures, including ResNets, results in a boost in accuracy and faster learning. A detailed analysis of our models reveals that they perform hierarchical compositionality by learning feature maps relative to the compressed representations provided by the skip connections.