Reverse Personalization

TL;DR

This paper introduces a reverse personalization framework for face anonymization using conditional diffusion inversion, enabling attribute-controlled anonymization without requiring model fine-tuning or well-represented subjects.

Contribution

It presents a novel method that allows direct, attribute-controllable face anonymization through diffusion model inversion and identity-guided conditioning.

Findings

Achieves state-of-the-art balance between identity removal and attribute preservation.

Operates without fine-tuning or reliance on well-represented subjects.

Supports attribute-controllable face anonymization.

Abstract

Recent text-to-image diffusion models have demonstrated remarkable generation of realistic facial images conditioned on textual prompts and human identities, enabling creating personalized facial imagery. However, existing prompt-based methods for removing or modifying identity-specific features rely either on the subject being well-represented in the pre-trained model or require model fine-tuning for specific identities. In this work, we analyze the identity generation process and introduce a reverse personalization framework for face anonymization. Our approach leverages conditional diffusion inversion, allowing direct manipulation of images without using text prompts. To generalize beyond subjects in the model's training data, we incorporate an identity-guided conditioning branch. Unlike prior anonymization methods, which lack control over facial attributes, our framework supports attribute-controllable anonymization. We demonstrate that our method achieves a state-of-the-art balance between identity removal, attribute preservation, and image quality. Source code and data are available at https://github.com/hanweikung/reverse-personalization .