MUSE: Manipulating Unified Framework for Synthesizing Emotions in Images via Test-Time Optimization

TL;DR

MUSE is a unified framework for synthesizing and editing emotional content in images using test-time optimization, improving emotional accuracy and diversity without additional training or datasets.

Contribution

It introduces the first unified approach for emotional image generation and editing that leverages test-time optimization, avoiding extra datasets and model updates.

Findings

Outperforms existing methods in emotional accuracy

Enhances semantic diversity in generated images

Balances content fidelity and emotional expression

Abstract

Images evoke emotions that profoundly influence perception, often prioritized over content. Current Image Emotional Synthesis (IES) approaches artificially separate generation and editing tasks, creating inefficiencies and limiting applications where these tasks naturally intertwine, such as therapeutic interventions or storytelling. In this work, we introduce MUSE, the first unified framework capable of both emotional generation and editing. By adopting a strategy conceptually aligned with Test-Time Scaling (TTS) that widely used in both LLM and diffusion model communities, it avoids the requirement for additional updating diffusion model and specialized emotional synthesis datasets. More specifically, MUSE addresses three key questions in emotional synthesis: (1) HOW to stably guide synthesis by leveraging an off-the-shelf emotion classifier with gradient-based optimization of emotional tokens; (2) WHEN to introduce emotional guidance by identifying the optimal timing using semantic similarity as a supervisory signal; and (3) WHICH emotion to guide synthesis through a multi-emotion loss that reduces interference from inherent and similar emotions. Experimental results show that MUSE performs favorably against all methods for both generation and editing, improving emotional accuracy and semantic diversity while maintaining an optimal balance between desired content, adherence to text prompts, and realistic emotional expression. It establishes a new paradigm for emotion synthesis.