Deciding How to Decide: Dynamic Routing in Artificial Neural Networks

TL;DR

This paper introduces and evaluates three strategies for training neural networks with dynamic routing, where different inputs follow different paths, leading to specialized processing and improved performance under fixed computational constraints.

Contribution

It systematically compares three dynamic routing strategies and demonstrates their effectiveness in creating specialized layers and improving image classification performance.

Findings

Layers and branches become category-specific.

Dynamically-routed networks outperform static ones at fixed computational budgets.

Different routing strategies have comparable qualitative network structures.

Abstract

We propose and systematically evaluate three strategies for training dynamically-routed artificial neural networks: graphs of learned transformations through which different input signals may take different paths. Though some approaches have advantages over others, the resulting networks are often qualitatively similar. We find that, in dynamically-routed networks trained to classify images, layers and branches become specialized to process distinct categories of images. Additionally, given a fixed computational budget, dynamically-routed networks tend to perform better than comparable statically-routed networks.