UniNet: Unified Architecture Search with Convolution, Transformer, and MLP

TL;DR

UniNet introduces a unified architecture search method that combines convolution, transformer, and MLP operators, along with novel context-aware down-sampling modules, to achieve superior performance across multiple vision tasks.

Contribution

This work presents the first unified search framework for combining convolution, transformer, and MLP operators with novel down-sampling modules for high-performance visual networks.

Findings

Outperforms EfficientNet and Swin-Transformer on ImageNet, COCO, and ADE20K.

Identifies down-sampling bottlenecks in combined architectures.

Proposes effective context-aware down-sampling modules.

Abstract

Recently, transformer and multi-layer perceptron (MLP) architectures have achieved impressive results on various vision tasks. A few works investigated manually combining those operators to design visual network architectures, and can achieve satisfactory performances to some extent. In this paper, we propose to jointly search the optimal combination of convolution, transformer, and MLP for building a series of all-operator network architectures with high performances on visual tasks. We empirically identify that the widely-used strided convolution or pooling based down-sampling modules become the performance bottlenecks when the operators are combined to form a network. To better tackle the global context captured by the transformer and MLP operators, we propose two novel context-aware down-sampling modules, which can better adapt to the global information encoded by transformer and MLP operators. To this end, we jointly search all operators and down-sampling modules in a unified search space. Notably, Our searched network UniNet (Unified Network) outperforms state-of-the-art pure convolution-based architecture, EfficientNet, and pure transformer-based architecture, Swin-Transformer, on multiple public visual benchmarks, ImageNet classification, COCO object detection, and ADE20K semantic segmentation.