Differentiable Neural Architecture Transformation for Reproducible Architecture Improvement

TL;DR

This paper introduces a differentiable neural architecture transformation method that enhances reproducibility and efficiency, outperforming previous NAT approaches across multiple datasets and architectures.

Contribution

The paper proposes a novel differentiable transformation technique that improves reproducibility and performance in neural architecture optimization.

Findings

Outperforms NAT in accuracy and stability

Demonstrates reproducibility across CIFAR-10 and Tiny ImageNet

Applicable to various neural architectures

Abstract

Recently, Neural Architecture Search (NAS) methods are introduced and show impressive performance on many benchmarks. Among those NAS studies, Neural Architecture Transformer (NAT) aims to improve the given neural architecture to have better performance while maintaining computational costs. However, NAT has limitations about a lack of reproducibility. In this paper, we propose differentiable neural architecture transformation that is reproducible and efficient. The proposed method shows stable performance on various architectures. Extensive reproducibility experiments on two datasets, i.e., CIFAR-10 and Tiny Imagenet, present that the proposed method definitely outperforms NAT and be applicable to other models and datasets.