Graph-based Neural Architecture Search with Operation Embeddings

TL;DR

This paper introduces learnable operation embeddings into neural architecture search, improving the encoding of architectures and leading to state-of-the-art results by capturing structural properties more effectively.

Contribution

It replaces fixed operator encodings with learnable representations, enhancing the accuracy and smoothness of architecture encoding in NAS.

Findings

Achieved state-of-the-art performance on ENAS benchmark

Generated architectures with similar operation and graph patterns

Demonstrated improved architecture representations

Abstract

Neural Architecture Search (NAS) has recently gained increased attention, as a class of approaches that automatically searches in an input space of network architectures. A crucial part of the NAS pipeline is the encoding of the architecture that consists of the applied computational blocks, namely the operations and the links between them. Most of the existing approaches either fail to capture the structural properties of the architectures or use hand-engineered vector to encode the operator information. In this paper, we propose the replacement of fixed operator encoding with learnable representations in the optimization process. This approach, which effectively captures the relations of different operations, leads to smoother and more accurate representations of the architectures and consequently to improved performance of the end task. Our extensive evaluation in ENAS benchmark demonstrates the effectiveness of the proposed operation embeddings to the generation of highly accurate models, achieving state-of-the-art performance. Finally, our method produces top-performing architectures that share similar operation and graph patterns, highlighting a strong correlation between the structural properties of the architecture and its performance.