PerTouch: VLM-Driven Agent for Personalized and Semantic Image Retouching

TL;DR

PerTouch is a diffusion-based framework that enables personalized, semantic-aware image retouching by integrating visual language models, semantic boundary handling, and user feedback mechanisms for improved control and aesthetic alignment.

Contribution

It introduces a unified diffusion model with semantic control, a VLM-driven agent for natural language interaction, and feedback mechanisms for personalized image retouching.

Findings

Outperforms existing methods in personalized retouching quality

Effective semantic boundary perception improves retouching precision

VLM-driven agent successfully interprets user instructions

Abstract

Image retouching aims to enhance visual quality while aligning with users' personalized aesthetic preferences. To address the challenge of balancing controllability and subjectivity, we propose a unified diffusion-based image retouching framework called PerTouch. Our method supports semantic-level image retouching while maintaining global aesthetics. Using parameter maps containing attribute values in specific semantic regions as input, PerTouch constructs an explicit parameter-to-image mapping for fine-grained image retouching. To improve semantic boundary perception, we introduce semantic replacement and parameter perturbation mechanisms during training. To connect natural language instructions with visual control, we develop a VLM-driven agent to handle both strong and weak user instructions. Equipped with mechanisms of feedback-driven rethinking and scene-aware memory, PerTouch better aligns with user intent and captures long-term preferences. Extensive experiments demonstrate each component's effectiveness and the superior performance of PerTouch in personalized image retouching. Code Pages: https://github.com/Auroral703/PerTouch.