DNArch: Learning Convolutional Neural Architectures by Backpropagation

TL;DR

DNArch introduces a differentiable method for jointly learning CNN weights and architectures, enabling automatic discovery of optimal configurations like kernel sizes, channels, downsampling, and depth, across various tasks.

Contribution

It proposes a novel continuous, differentiable approach to neural architecture search that is not limited to predefined components, allowing comprehensive architecture discovery.

Findings

DNArch effectively finds high-performing CNN architectures for classification and dense prediction tasks.

It can incorporate computational budget constraints during training.

Demonstrates versatility across sequential and image data tasks.

Abstract

We present Differentiable Neural Architectures (DNArch), a method that jointly learns the weights and the architecture of Convolutional Neural Networks (CNNs) by backpropagation. In particular, DNArch allows learning (i) the size of convolutional kernels at each layer, (ii) the number of channels at each layer, (iii) the position and values of downsampling layers, and (iv) the depth of the network. To this end, DNArch views neural architectures as continuous multidimensional entities, and uses learnable differentiable masks along each dimension to control their size. Unlike existing methods, DNArch is not limited to a predefined set of possible neural components, but instead it is able to discover entire CNN architectures across all feasible combinations of kernel sizes, widths, depths and downsampling. Empirically, DNArch finds performant CNN architectures for several classification and dense prediction tasks on sequential and image data. When combined with a loss term that controls the network complexity, DNArch constrains its search to architectures that respect a predefined computational budget during training.