StylePrompter: All Styles Need Is Attention

TL;DR

StylePrompter introduces a Transformer-based approach for GAN inversion that enhances image reconstruction and editing flexibility by leveraging hierarchical vision Transformers and style-driven refinement.

Contribution

It pioneers the use of a hierarchical vision Transformer backbone and a Style-driven Multi-scale Adaptive Refinement Transformer for improved GAN inversion and editing.

Findings

Achieves high-quality image inversion with balanced reconstruction and editability.

Outperforms existing methods involving $ ext{F}$ space in inversion tasks.

Demonstrates adaptability to various image editing tasks.

Abstract

GAN inversion aims at inverting given images into corresponding latent codes for Generative Adversarial Networks (GANs), especially StyleGAN where exists a disentangled latent space that allows attribute-based image manipulation at latent level. As most inversion methods build upon Convolutional Neural Networks (CNNs), we transfer a hierarchical vision Transformer backbone innovatively to predict $\mathcal{W^+}$ latent codes at token level. We further apply a Style-driven Multi-scale Adaptive Refinement Transformer (SMART) in $\mathcal{F}$ space to refine the intermediate style features of the generator. By treating style features as queries to retrieve lost identity information from the encoder's feature maps, SMART can not only produce high-quality inverted images but also surprisingly adapt to editing tasks. We then prove that StylePrompter lies in a more disentangled $\mathcal{W^+}$ and show the controllability of SMART. Finally, quantitative and qualitative experiments demonstrate that StylePrompter can achieve desirable performance in balancing reconstruction quality and editability, and is "smart" enough to fit into most edits, outperforming other $\mathcal{F}$-involved inversion methods.