Neural Architecture Search via Bregman Iterations

TL;DR

This paper introduces a new neural architecture search method using Bregman iterations, enabling the automatic design of task-specific neural networks by gradually adding relevant parameters.

Contribution

It presents a gradient-based one-shot NAS algorithm that employs Bregman iterations to efficiently discover optimal architectures in an inverse scale space manner.

Findings

Unveils residual autoencoders for denoising, deblurring, and classification.

Demonstrates effective architecture search via Bregman iteration.

Provides open-source code for implementation.

Abstract

We propose a novel strategy for Neural Architecture Search (NAS) based on Bregman iterations. Starting from a sparse neural network our gradient-based one-shot algorithm gradually adds relevant parameters in an inverse scale space manner. This allows the network to choose the best architecture in the search space which makes it well-designed for a given task, e.g., by adding neurons or skip connections. We demonstrate that using our approach one can unveil, for instance, residual autoencoders for denoising, deblurring, and classification tasks. Code is available at https://github.com/TimRoith/BregmanLearning.