What Decreases Editing Capability? Domain-Specific Hybrid Refinement for Improved GAN Inversion

TL;DR

This paper introduces Domain-Specific Hybrid Refinement (DHR), a novel method that improves GAN inversion fidelity while preserving editing capabilities, especially for complex images, by segmenting images and applying tailored refinement techniques.

Contribution

The paper proposes a new hybrid refinement approach combining segmentation and modulation techniques to enhance inversion fidelity without sacrificing editability.

Findings

Achieves state-of-the-art results in real image inversion.

Maintains editing capability on complex images.

Compatible with all latent code embedding methods.

Abstract

Recently, inversion methods have focused on additional high-rate information in the generator (e.g., weights or intermediate features) to refine inversion and editing results from embedded latent codes. Although these techniques gain reasonable improvement in reconstruction, they decrease editing capability, especially on complex images (e.g., containing occlusions, detailed backgrounds, and artifacts). A vital crux is refining inversion results, avoiding editing capability degradation. To tackle this problem, we introduce Domain-Specific Hybrid Refinement (DHR), which draws on the advantages and disadvantages of two mainstream refinement techniques to maintain editing ability with fidelity improvement. Specifically, we first propose Domain-Specific Segmentation to segment images into two parts: in-domain and out-of-domain parts. The refinement process aims to maintain the editability for in-domain areas and improve two domains' fidelity. We refine these two parts by weight modulation and feature modulation, which we call Hybrid Modulation Refinement. Our proposed method is compatible with all latent code embedding methods. Extension experiments demonstrate that our approach achieves state-of-the-art in real image inversion and editing. Code is available at https://github.com/caopulan/Domain-Specific_Hybrid_Refinement_Inversion.