Growth strategies for arbitrary DAG neural architectures

TL;DR

This paper proposes methods for dynamically growing arbitrary DAG neural architectures during training to improve efficiency and reduce costs, by expanding models in a flexible, data-driven manner.

Contribution

It extends neural architecture growth techniques to arbitrary DAGs, enabling more efficient and adaptable network expansion during training.

Findings

Effective strategies for reducing computational costs.

Demonstrated growth of DAG architectures during training.

Improved parameter efficiency in neural networks.

Abstract

Deep learning has shown impressive results obtained at the cost of training huge neural networks. However, the larger the architecture, the higher the computational, financial, and environmental costs during training and inference. We aim at reducing both training and inference durations. We focus on Neural Architecture Growth, which can increase the size of a small model when needed, directly during training using information from the backpropagation. We expand existing work and freely grow neural networks in the form of any Directed Acyclic Graph by reducing expressivity bottlenecks in the architecture. We explore strategies to reduce excessive computations and steer network growth toward more parameter-efficient architectures.