FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editing

TL;DR

FACEMUG is a novel multimodal framework for local facial editing that supports diverse input modalities, maintains image quality after multiple edits, and enables fine-grained, semantic, and globally consistent facial manipulations.

Contribution

It introduces a unified multimodal generative model with a novel fusion mechanism and a self-supervised latent warping algorithm for improved local facial editing.

Findings

Outperforms state-of-the-art methods in editing quality and flexibility.

Supports multiple input modalities including sketches, text, and attribute labels.

Maintains high image quality after multiple incremental edits.

Abstract

Existing facial editing methods have achieved remarkable results, yet they often fall short in supporting multimodal conditional local facial editing. One of the significant evidences is that their output image quality degrades dramatically after several iterations of incremental editing, as they do not support local editing. In this paper, we present a novel multimodal generative and fusion framework for globally-consistent local facial editing (FACEMUG) that can handle a wide range of input modalities and enable fine-grained and semantic manipulation while remaining unedited parts unchanged. Different modalities, including sketches, semantic maps, color maps, exemplar images, text, and attribute labels, are adept at conveying diverse conditioning details, and their combined synergy can provide more explicit guidance for the editing process. We thus integrate all modalities into a unified generative latent space to enable multimodal local facial edits. Specifically, a novel multimodal feature fusion mechanism is proposed by utilizing multimodal aggregation and style fusion blocks to fuse facial priors and multimodalities in both latent and feature spaces. We further introduce a novel self-supervised latent warping algorithm to rectify misaligned facial features, efficiently transferring the pose of the edited image to the given latent codes. We evaluate our FACEMUG through extensive experiments and comparisons to state-of-the-art (SOTA) methods. The results demonstrate the superiority of FACEMUG in terms of editing quality, flexibility, and semantic control, making it a promising solution for a wide range of local facial editing tasks.