Towards Localized Fine-Grained Control for Facial Expression Generation

TL;DR

This paper introduces a method using Action Units (AUs) for precise, localized control of facial expressions in generative models, enabling the creation of authentic and nuanced facial reactions beyond stereotypical expressions.

Contribution

It presents a novel approach to facial expression control in generative models using AUs, allowing for detailed and unconventional expression synthesis.

Findings

Enables generation of authentic, nuanced facial expressions.

Allows integration with text and image prompts for precise control.

Supports creation of unconventional facial expressions.

Abstract

Generative models have surged in popularity recently due to their ability to produce high-quality images and video. However, steering these models to produce images with specific attributes and precise control remains challenging. Humans, particularly their faces, are central to content generation due to their ability to convey rich expressions and intent. Current generative models mostly generate flat neutral expressions and characterless smiles without authenticity. Other basic expressions like anger are possible, but are limited to the stereotypical expression, while other unconventional facial expressions like doubtful are difficult to reliably generate. In this work, we propose the use of AUs (action units) for facial expression control in face generation. AUs describe individual facial muscle movements based on facial anatomy, allowing precise and localized control over the intensity of facial movements. By combining different action units, we unlock the ability to create unconventional facial expressions that go beyond typical emotional models, enabling nuanced and authentic reactions reflective of real-world expressions. The proposed method can be seamlessly integrated with both text and image prompts using adapters, offering precise and intuitive control of the generated results. Code and dataset are available in {https://github.com/tvaranka/fineface}.