Conflicting Bundles: Adapting Architectures Towards the Improved Training of Deep Neural Networks

TL;DR

This paper introduces a new theory and metric to identify and remove conflicting layers in neural networks early in training, improving performance, reducing memory, and speeding up inference.

Contribution

It presents a novel theoretical framework and algorithm for automatically detecting and removing performance-degrading layers in neural networks.

Findings

Identified layers that worsen performance due to conflicting training bundles

Proposed an algorithm that automatically removes such layers

Resulting architectures achieve competitive accuracy with reduced memory and faster inference

Abstract

Designing neural network architectures is a challenging task and knowing which specific layers of a model must be adapted to improve the performance is almost a mystery. In this paper, we introduce a novel theory and metric to identify layers that decrease the test accuracy of the trained models, this identification is done as early as at the beginning of training. In the worst-case, such a layer could lead to a network that can not be trained at all. More precisely, we identified those layers that worsen the performance because they produce conflicting training bundles as we show in our novel theoretical analysis, complemented by our extensive empirical studies. Based on these findings, a novel algorithm is introduced to remove performance decreasing layers automatically. Architectures found by this algorithm achieve a competitive accuracy when compared against the state-of-the-art architectures. While keeping such high accuracy, our approach drastically reduces memory consumption and inference time for different computer vision tasks.