Controlled Face Manipulation and Synthesis for Data Augmentation

TL;DR

This paper introduces a controllable face editing method in the latent space of a pre-trained generator to improve facial expression analysis, addressing label scarcity and attribute entanglement.

Contribution

It proposes a novel disentanglement approach using dependency-aware conditioning and orthogonal projection for precise AU manipulation in face images.

Findings

Enhanced AU detection accuracy with augmented data

Fewer artifacts and better identity preservation in edits

Outperforms existing methods in disentanglement and realism

Abstract

Deep learning vision models excel with abundant supervision, but many applications face label scarcity and class imbalance. Controllable image editing can augment scarce labeled data, yet edits often introduce artifacts and entangle non-target attributes. We study this in facial expression analysis, targeting Action Unit (AU) manipulation where annotation is costly and AU co-activation drives entanglement. We present a facial manipulation method that operates in the semantic latent space of a pre-trained face generator (Diffusion Autoencoder). Using lightweight linear models, we reduce entanglement of semantic features via (i) dependency-aware conditioning that accounts for AU co-activation, and (ii) orthogonal projection that removes nuisance attribute directions (e.g., glasses), together with an expression neutralization step to enable absolute AU edit. We use these edits to balance AU occurrence by editing labeled faces and to diversify identities/demographics via controlled synthesis. Augmenting AU detector training with the generated data improves accuracy and yields more disentangled predictions with fewer co-activation shortcuts, outperforming alternative data-efficient training strategies and suggesting improvements similar to what would require substantially more labeled data in our learning-curve analysis. Compared to prior methods, our edits are stronger, produce fewer artifacts, and preserve identity better.