CONetV2: Efficient Auto-Channel Size Optimization for CNNs

TL;DR

CONetV2 presents an efficient method for optimizing CNN channel sizes using dependency analysis, knowledge distillation, and a novel performance metric, achieving superior architectures with less computational cost.

Contribution

The paper introduces a novel, resource-efficient NAS approach focusing on micro-search space of channel sizes, integrating dependency extraction, a new performance metric, and knowledge distillation.

Findings

Outperforms baseline architectures significantly.

Reduces computational resources required for NAS.

Provides a new metric correlating with test accuracy.

Abstract

Neural Architecture Search (NAS) has been pivotal in finding optimal network configurations for Convolution Neural Networks (CNNs). While many methods explore NAS from a global search-space perspective, the employed optimization schemes typically require heavy computational resources. This work introduces a method that is efficient in computationally constrained environments by examining the micro-search space of channel size. In tackling channel-size optimization, we design an automated algorithm to extract the dependencies within different connected layers of the network. In addition, we introduce the idea of knowledge distillation, which enables preservation of trained weights, admist trials where the channel sizes are changing. Further, since the standard performance indicators (accuracy, loss) fail to capture the performance of individual network components (providing an overall network evaluation), we introduce a novel metric that highly correlates with test accuracy and enables analysis of individual network layers. Combining dependency extraction, metrics, and knowledge distillation, we introduce an efficient searching algorithm, with simulated annealing inspired stochasticity, and demonstrate its effectiveness in finding optimal architectures that outperform baselines by a large margin.