Towards LLM-centric Affective Visual Customization via Efficient and Precise Emotion Manipulating

TL;DR

This paper introduces an LLM-based affective visual customization framework that efficiently and accurately modifies images' subjective emotions while preserving content, addressing limitations of prior methods that ignored emotional content.

Contribution

It proposes a novel LLM-centric affective visual customization task and introduces the EPEM approach with modules for efficient emotion conversion and precise content retention.

Findings

EPEM outperforms state-of-the-art baselines in experiments

The approach effectively manipulates subjective emotions in images

The method preserves emotion-agnostic content accurately

Abstract

Previous studies on visual customization primarily rely on the objective alignment between various control signals (e.g., language, layout and canny) and the edited images, which largely ignore the subjective emotional contents, and more importantly lack general-purpose foundation models for affective visual customization. With this in mind, this paper proposes an LLM-centric Affective Visual Customization (L-AVC) task, which focuses on generating images within modifying their subjective emotions via Multimodal LLM. Further, this paper contends that how to make the model efficiently align emotion conversion in semantics (named inter-emotion semantic conversion) and how to precisely retain emotion-agnostic contents (named exter-emotion semantic retaining) are rather important and challenging in this L-AVC task. To this end, this paper proposes an Efficient and Precise Emotion Manipulating approach for editing subjective emotions in images. Specifically, an Efficient Inter-emotion Converting (EIC) module is tailored to make the LLM efficiently align emotion conversion in semantics before and after editing, followed by a Precise Exter-emotion Retaining (PER) module to precisely retain the emotion-agnostic contents. Comprehensive experimental evaluations on our constructed L-AVC dataset demonstrate the great advantage of the proposed EPEM approach to the L-AVC task over several state-of-the-art baselines. This justifies the importance of emotion information for L-AVC and the effectiveness of EPEM in efficiently and precisely manipulating such information.