Auto DragGAN: Editing the Generative Image Manifold in an Autoregressive Manner

TL;DR

Auto DragGAN introduces a novel autoregressive approach using a transformer-based network to enable fast, pixel-level precise image editing by predicting latent code movements, outperforming previous methods in speed and accuracy.

Contribution

This work is the first to employ a regression-based network for learning StyleGAN latent variations during image dragging, achieving high-fidelity, pixel-level editing with real-time inference.

Findings

Achieves state-of-the-art inference speed and editing quality.

Predicts small pixel movements for high-fidelity edits.

Uses autoregressive latent trajectory prediction for stability.

Abstract

Pixel-level fine-grained image editing remains an open challenge. Previous works fail to achieve an ideal trade-off between control granularity and inference speed. They either fail to achieve pixel-level fine-grained control, or their inference speed requires optimization. To address this, this paper for the first time employs a regression-based network to learn the variation patterns of StyleGAN latent codes during the image dragging process. This method enables pixel-level precision in dragging editing with little time cost. Users can specify handle points and their corresponding target points on any GAN-generated images, and our method will move each handle point to its corresponding target point. Through experimental analysis, we discover that a short movement distance from handle points to target points yields a high-fidelity edited image, as the model only needs to predict the movement of a small portion of pixels. To achieve this, we decompose the entire movement process into multiple sub-processes. Specifically, we develop a transformer encoder-decoder based network named 'Latent Predictor' to predict the latent code motion trajectories from handle points to target points in an autoregressive manner. Moreover, to enhance the prediction stability, we introduce a component named 'Latent Regularizer', aimed at constraining the latent code motion within the distribution of natural images. Extensive experiments demonstrate that our method achieves state-of-the-art (SOTA) inference speed and image editing performance at the pixel-level granularity.