Auto-tuning of Deep Neural Networks by Conflicting Layer Removal

TL;DR

This paper introduces a method to identify and remove conflicting layers in neural networks, leading to simpler architectures with comparable accuracy and reduced resource consumption.

Contribution

It presents a novel approach to detect conflicting layers early in training and a NAS algorithm to optimize network architecture by removing such layers.

Findings

Up to 60% of layers can be removed without significant accuracy loss.

The proposed NAS algorithm effectively identifies conflicting layers early.

Architectures with removed conflicting layers achieve competitive accuracy with less memory and faster inference.

Abstract

Designing neural network architectures is a challenging task and knowing which specific layers of a model must be adapted to improve the performance is almost a mystery. In this paper, we introduce a novel methodology to identify layers that decrease the test accuracy of trained models. Conflicting layers are detected as early as the beginning of training. In the worst-case scenario, we prove that such a layer could lead to a network that cannot be trained at all. A theoretical analysis is provided on what is the origin of those layers that result in a lower overall network performance, which is complemented by our extensive empirical evaluation. More precisely, we identified those layers that worsen the performance because they would produce what we name conflicting training bundles. We will show that around 60% of the layers of trained residual networks can be completely removed from the architecture with no significant increase in the test-error. We will further present a novel neural-architecture-search (NAS) algorithm that identifies conflicting layers at the beginning of the training. Architectures found by our auto-tuning algorithm achieve competitive accuracy values when compared against more complex state-of-the-art architectures, while drastically reducing memory consumption and inference time for different computer vision tasks. The source code is available on https://github.com/peerdavid/conflicting-bundles