Deep Feature Surgery: Towards Accurate and Efficient Multi-Exit Networks

TL;DR

Deep Feature Surgery ( extbackslash methodname) improves multi-exit networks by resolving gradient conflicts through feature partitioning and referencing, leading to faster training and higher accuracy across datasets.

Contribution

The paper introduces Deep Feature Surgery, a novel method that alleviates gradient conflicts in multi-exit networks, enhancing training efficiency and model accuracy.

Findings

Up to 50 extbackslash extbf{0} extbackslash extbf{ extbackslash 0} reduction in training time.

Up to 6.94 extbackslash extbf{ extbackslash 0} accuracy improvement.

Uses about 2 times fewer FLOPs to achieve same accuracy.

Abstract

Multi-exit network is a promising architecture for efficient model inference by sharing backbone networks and weights among multiple exits. However, the gradient conflict of the shared weights results in sub-optimal accuracy. This paper introduces Deep Feature Surgery (\methodname), which consists of feature partitioning and feature referencing approaches to resolve gradient conflict issues during the training of multi-exit networks. The feature partitioning separates shared features along the depth axis among all exits to alleviate gradient conflict while simultaneously promoting joint optimization for each exit. Subsequently, feature referencing enhances multi-scale features for distinct exits across varying depths to improve the model accuracy. Furthermore, \methodname~reduces the training operations with the reduced complexity of backpropagation. Experimental results on Cifar100 and ImageNet datasets exhibit that \methodname~provides up to a \textbf{50.00\%} reduction in training time and attains up to a \textbf{6.94\%} enhancement in accuracy when contrasted with baseline methods across diverse models and tasks. Budgeted batch classification evaluation on MSDNet demonstrates that DFS uses about $\mathbf{2}\boldsymbol{\times}$ fewer average FLOPs per image to achieve the same classification accuracy as baseline methods on Cifar100. The code is available at https://github.com/GongCheng1919/dfs.