Network Fission Ensembles for Low-Cost Self-Ensembles

TL;DR

Network Fission Ensembles (NFE) transforms a single neural network into a multi-exit structure through pruning and grouping, enabling low-cost ensemble inference without additional models, improving accuracy efficiently.

Contribution

The paper introduces Network Fission Ensembles, a novel method that creates multi-exit networks from a single model for cost-effective ensemble learning.

Findings

Achieves significant accuracy improvements over existing ensemble methods.

No additional computational burden during inference due to multi-exit structure.

Enhances performance through regularization from multiple exit losses.

Abstract

Recent ensemble learning methods for image classification have been shown to improve classification accuracy with low extra cost. However, they still require multiple trained models for ensemble inference, which eventually becomes a significant burden when the model size increases. In this paper, we propose a low-cost ensemble learning and inference, called Network Fission Ensembles (NFE), by converting a conventional network itself into a multi-exit structure. Starting from a given initial network, we first prune some of the weights to reduce the training burden. We then group the remaining weights into several sets and create multiple auxiliary paths using each set to construct multi-exits. We call this process Network Fission. Through this, multiple outputs can be obtained from a single network, which enables ensemble learning. Since this process simply changes the existing network structure to multi-exits without using additional networks, there is no extra computational burden for ensemble learning and inference. Moreover, by learning from multiple losses of all exits, the multi-exits improve performance via regularization, and high performance can be achieved even with increased network sparsity. With our simple yet effective method, we achieve significant improvement compared to existing ensemble methods. The code is available at https://github.com/hjdw2/NFE.