DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation

TL;DR

DynaST introduces a dynamic sparse attention Transformer that achieves fine-grained, efficient exemplar-guided image generation by adaptively focusing on relevant tokens, outperforming existing methods in detail quality and computational cost.

Contribution

The paper proposes a novel dynamic attention mechanism within a Transformer for flexible, fine-level matching in exemplar-guided image generation, applicable to both supervised and unsupervised tasks.

Findings

Outperforms state-of-the-art in local detail preservation

Reduces computational cost significantly

Effective across multiple image translation applications

Abstract

One key challenge of exemplar-guided image generation lies in establishing fine-grained correspondences between input and guided images. Prior approaches, despite the promising results, have relied on either estimating dense attention to compute per-point matching, which is limited to only coarse scales due to the quadratic memory cost, or fixing the number of correspondences to achieve linear complexity, which lacks flexibility. In this paper, we propose a dynamic sparse attention based Transformer model, termed Dynamic Sparse Transformer (DynaST), to achieve fine-level matching with favorable efficiency. The heart of our approach is a novel dynamic-attention unit, dedicated to covering the variation on the optimal number of tokens one position should focus on. Specifically, DynaST leverages the multi-layer nature of Transformer structure, and performs the dynamic attention scheme in a cascaded manner to refine matching results and synthesize visually-pleasing outputs. In addition, we introduce a unified training objective for DynaST, making it a versatile reference-based image translation framework for both supervised and unsupervised scenarios. Extensive experiments on three applications, pose-guided person image generation, edge-based face synthesis, and undistorted image style transfer, demonstrate that DynaST achieves superior performance in local details, outperforming the state of the art while reducing the computational cost significantly. Our code is available at https://github.com/Huage001/DynaST