HS-Diffusion: Semantic-Mixing Diffusion for Head Swapping

TL;DR

HS-Diffusion introduces a novel semantic-mixing diffusion approach for head swapping that preserves head and body integrity, achieves seamless transitions, and is validated on a new benchmark with superior results.

Contribution

The paper proposes a new semantic-mixing diffusion model for head swapping, including a semantic layout generator and a semantic calibration strategy, addressing the lack of datasets and benchmarks.

Findings

Outperforms existing methods on the new head swapping benchmark.

Effectively preserves head and body details with high-quality reconstruction.

Demonstrates superior seamlessness and realism in head swapping results.

Abstract

Image-based head swapping task aims to stitch a source head to another source body flawlessly. This seldom-studied task faces two major challenges: 1) Preserving the head and body from various sources while generating a seamless transition region. 2) No paired head swapping dataset and benchmark so far. In this paper, we propose a semantic-mixing diffusion model for head swapping (HS-Diffusion) which consists of a latent diffusion model (LDM) and a semantic layout generator. We blend the semantic layouts of source head and source body, and then inpaint the transition region by the semantic layout generator, achieving a coarse-grained head swapping. Semantic-mixing LDM can further implement a fine-grained head swapping with the inpainted layout as condition by a progressive fusion process, while preserving head and body with high-quality reconstruction. To this end, we propose a semantic calibration strategy for natural inpainting and a neck alignment for geometric realism. Importantly, we construct a new image-based head swapping benchmark and design two tailor-designed metrics (Mask-FID and Focal-FID). Extensive experiments demonstrate the superiority of our framework. The code will be available: https://github.com/qinghew/HS-Diffusion.