Learning Morphisms with Gauss-Newton Approximation for Growing Networks

TL;DR

This paper introduces a NAS method that uses Gauss-Newton approximation to efficiently grow neural networks, achieving comparable or better results with less computation on CIFAR datasets.

Contribution

We propose a novel NAS approach leveraging Gauss-Newton approximation for effective network growth, reducing computational cost compared to existing methods.

Findings

Achieves similar or better accuracy than state-of-the-art NAS methods.

Reduces computational cost for network growth.

Effective for CIFAR-10 and CIFAR-100 classification tasks.

Abstract

A popular method for Neural Architecture Search (NAS) is based on growing networks via small local changes to the network's architecture called network morphisms. These methods start with a small seed network and progressively grow the network by adding new neurons in an automated way. However, it remains a challenge to efficiently determine which parts of the network are best to grow. Here we propose a NAS method for growing a network by using a Gauss-Newton approximation of the loss function to efficiently learn and evaluate candidate network morphisms. We compare our method with state of the art NAS methods for CIFAR-10 and CIFAR-100 classification tasks, and conclude our method learns similar quality or better architectures at a smaller computational cost.