STAR: Synthesis of Tailored Architectures

TL;DR

STAR introduces a novel evolutionary approach for synthesizing tailored neural network architectures by leveraging a new search space based on linear input-varying systems theory, enabling efficient multi-metric optimization.

Contribution

The paper presents a new architecture synthesis method combining a theoretically grounded search space with evolutionary algorithms for multi-objective optimization.

Findings

Outperforms existing Transformer models in quality and efficiency.

Enables optimization of diverse architectures with various computational units.

Achieves state-of-the-art results in autoregressive language modeling.

Abstract

Iterative improvement of model architectures is fundamental to deep learning: Transformers first enabled scaling, and recent advances in model hybridization have pushed the quality-efficiency frontier. However, optimizing architectures remains challenging and expensive. Current automated or manual approaches fall short, largely due to limited progress in the design of search spaces and due to the simplicity of resulting patterns and heuristics. In this work, we propose a new approach for the synthesis of tailored architectures (STAR). Our approach combines a novel search space based on the theory of linear input-varying systems, supporting a hierarchical numerical encoding into architecture genomes. STAR genomes are automatically refined and recombined with gradient-free, evolutionary algorithms to optimize for multiple model quality and efficiency metrics. Using STAR, we optimize large populations of new architectures, leveraging diverse computational units and interconnection patterns, improving over highly-optimized Transformers and striped hybrid models on the frontier of quality, parameter size, and inference cache for autoregressive language modeling.