IC-Portrait: In-Context Matching for View-Consistent Personalized Portrait

TL;DR

IC-Portrait introduces a novel framework that leverages in-context dense correspondence matching in diffusion models to generate personalized, view-consistent portraits with enhanced identity preservation and relighting capabilities.

Contribution

The paper proposes a new approach combining lighting-aware stitching and view-consistent adaptation to improve personalized portrait generation in diffusion models.

Findings

Outperforms state-of-the-art methods quantitatively and qualitatively.

Achieves significant improvements in visual quality and identity fidelity.

Demonstrates 3D-aware relighting capabilities.

Abstract

Existing diffusion models show great potential for identity-preserving generation. However, personalized portrait generation remains challenging due to the diversity in user profiles, including variations in appearance and lighting conditions. To address these challenges, we propose IC-Portrait, a novel framework designed to accurately encode individual identities for personalized portrait generation. Our key insight is that pre-trained diffusion models are fast learners (e.g.,100 ~ 200 steps) for in-context dense correspondence matching, which motivates the two major designs of our IC-Portrait framework. Specifically, we reformulate portrait generation into two sub-tasks: 1) Lighting-Aware Stitching: we find that masking a high proportion of the input image, e.g., 80%, yields a highly effective self-supervisory representation learning of reference image lighting. 2) View-Consistent Adaptation: we leverage a synthetic view-consistent profile dataset to learn the in-context correspondence. The reference profile can then be warped into arbitrary poses for strong spatial-aligned view conditioning. Coupling these two designs by simply concatenating latents to form ControlNet-like supervision and modeling, enables us to significantly enhance the identity preservation fidelity and stability. Extensive evaluations demonstrate that IC-Portrait consistently outperforms existing state-of-the-art methods both quantitatively and qualitatively, with particularly notable improvements in visual qualities. Furthermore, IC-Portrait even demonstrates 3D-aware relighting capabilities.