AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation

TL;DR

AsCAN introduces an asymmetric hybrid convolution-transformer architecture that achieves efficient recognition and generation across multiple tasks, offering superior performance-latency trade-offs and state-of-the-art results in large-scale text-to-image synthesis.

Contribution

The paper proposes a novel asymmetric hybrid architecture combining convolutional and transformer blocks, optimized for diverse tasks and scalable to large-scale applications.

Findings

Supports multiple tasks including recognition, segmentation, and image generation.

Achieves faster inference speed than existing efficient transformer models.

Sets new state-of-the-art performance in large-scale text-to-image synthesis.

Abstract

Neural network architecture design requires making many crucial decisions. The common desiderata is that similar decisions, with little modifications, can be reused in a variety of tasks and applications. To satisfy that, architectures must provide promising latency and performance trade-offs, support a variety of tasks, scale efficiently with respect to the amounts of data and compute, leverage available data from other tasks, and efficiently support various hardware. To this end, we introduce AsCAN -- a hybrid architecture, combining both convolutional and transformer blocks. We revisit the key design principles of hybrid architectures and propose a simple and effective \emph{asymmetric} architecture, where the distribution of convolutional and transformer blocks is \emph{asymmetric}, containing more convolutional blocks in the earlier stages, followed by more transformer blocks in later stages. AsCAN supports a variety of tasks: recognition, segmentation, class-conditional image generation, and features a superior trade-off between performance and latency. We then scale the same architecture to solve a large-scale text-to-image task and show state-of-the-art performance compared to the most recent public and commercial models. Notably, even without any computation optimization for transformer blocks, our models still yield faster inference speed than existing works featuring efficient attention mechanisms, highlighting the advantages and the value of our approach.