Connectivity Learning in Multi-Branch Networks

TL;DR

This paper introduces a method to automatically learn the optimal connectivity between branches in multi-branch neural networks, leading to improved classification accuracy over fixed-connection architectures.

Contribution

It proposes an algorithm that learns multi-branch network connections jointly with weights, removing predefined design choices and optimizing for the end task.

Findings

Achieves higher accuracy than ResNeXt on multiple datasets.

Automatically learns branch connectivity, reducing manual design effort.

Improves performance without increasing model capacity.

Abstract

While much of the work in the design of convolutional networks over the last five years has revolved around the empirical investigation of the importance of depth, filter sizes, and number of feature channels, recent studies have shown that branching, i.e., splitting the computation along parallel but distinct threads and then aggregating their outputs, represents a new promising dimension for significant improvements in performance. To combat the complexity of design choices in multi-branch architectures, prior work has adopted simple strategies, such as a fixed branching factor, the same input being fed to all parallel branches, and an additive combination of the outputs produced by all branches at aggregation points. In this work we remove these predefined choices and propose an algorithm to learn the connections between branches in the network. Instead of being chosen a priori by the human designer, the multi-branch connectivity is learned simultaneously with the weights of the network by optimizing a single loss function defined with respect to the end task. We demonstrate our approach on the problem of multi-class image classification using three different datasets where it yields consistently higher accuracy compared to the state-of-the-art "ResNeXt" multi-branch network given the same learning capacity.