GroupDiff: Diffusion-based Group Portrait Editing

TL;DR

GroupDiff introduces a diffusion-based method for flexible, high-fidelity group portrait editing, addressing data scarcity and ensuring appearance consistency through novel data generation and attention-guided techniques.

Contribution

It presents a comprehensive framework with a data engine, appearance preservation, and control mechanisms for effective group photo editing.

Findings

Achieves state-of-the-art performance in group portrait editing.

Maintains appearance consistency after editing.

Provides flexible control over editing operations.

Abstract

Group portrait editing is highly desirable since users constantly want to add a person, delete a person, or manipulate existing persons. It is also challenging due to the intricate dynamics of human interactions and the diverse gestures. In this work, we present GroupDiff, a pioneering effort to tackle group photo editing with three dedicated contributions: 1) Data Engine: Since there is no labeled data for group photo editing, we create a data engine to generate paired data for training. The training data engine covers the diverse needs of group portrait editing. 2) Appearance Preservation: To keep the appearance consistent after editing, we inject the images of persons from the group photo into the attention modules and employ skeletons to provide intra-person guidance. 3) Control Flexibility: Bounding boxes indicating the locations of each person are used to reweight the attention matrix so that the features of each person can be injected into the correct places. This inter-person guidance provides flexible manners for manipulation. Extensive experiments demonstrate that GroupDiff exhibits state-of-the-art performance compared to existing methods. GroupDiff offers controllability for editing and maintains the fidelity of the original photos.