DASViT: Differentiable Architecture Search for Vision Transformer

TL;DR

DASViT introduces a differentiable neural architecture search method tailored for Vision Transformers, enabling the discovery of innovative, efficient architectures that outperform traditional designs with fewer parameters and FLOPs.

Contribution

This paper presents DASViT, a novel differentiable NAS approach specifically designed for ViTs, addressing limitations of previous methods and enabling the discovery of superior architectures.

Findings

DASViT architectures outperform ViT-B/16 on multiple datasets.

DASViT designs are more efficient with fewer parameters and FLOPs.

DASViT uncovers innovative Transformer architectures beyond traditional designs.

Abstract

Designing effective neural networks is a cornerstone of deep learning, and Neural Architecture Search (NAS) has emerged as a powerful tool for automating this process. Among the existing NAS approaches, Differentiable Architecture Search (DARTS) has gained prominence for its efficiency and ease of use, inspiring numerous advancements. Since the rise of Vision Transformers (ViT), researchers have applied NAS to explore ViT architectures, often focusing on macro-level search spaces and relying on discrete methods like evolutionary algorithms. While these methods ensure reliability, they face challenges in discovering innovative architectural designs, demand extensive computational resources, and are time-intensive. To address these limitations, we introduce Differentiable Architecture Search for Vision Transformer (DASViT), which bridges the gap in differentiable search for ViTs and uncovers novel designs. Experiments show that DASViT delivers architectures that break traditional Transformer encoder designs, outperform ViT-B/16 on multiple datasets, and achieve superior efficiency with fewer parameters and FLOPs.