A Closer Look at Branch Classifiers of Multi-exit Architectures

TL;DR

This paper investigates various branch classifier configurations in multi-exit neural networks, revealing that complexity-decreasing branching optimizes the accuracy-cost balance by minimally disrupting feature hierarchies.

Contribution

It provides a comprehensive analysis of branching patterns in multi-exit architectures and identifies complexity-decreasing branching as the most effective configuration.

Findings

Complexity-decreasing branches outperform others in accuracy-cost trade-off.

Least disruption to feature hierarchy occurs with complexity-decreasing branching.

Knowledge consistency analysis explains the effectiveness of complexity-decreasing branches.

Abstract

Multi-exit architectures consist of a backbone and branch classifiers that offer shortened inference pathways to reduce the run-time of deep neural networks. In this paper, we analyze different branching patterns that vary in their allocation of computational complexity for the branch classifiers. Constant-complexity branching keeps all branches the same, while complexity-increasing and complexity-decreasing branching place more complex branches later or earlier in the backbone respectively. Through extensive experimentation on multiple backbones and datasets, we find that complexity-decreasing branches are more effective than constant-complexity or complexity-increasing branches, which achieve the best accuracy-cost trade-off. We investigate a cause by using knowledge consistency to probe the effect of adding branches onto a backbone. Our findings show that complexity-decreasing branching yields the least disruption to the feature abstraction hierarchy of the backbone, which explains the effectiveness of the branching patterns.