Pareto-Enhanced Portrait Generation: Vision-Aligned Text Supervision for Alignment, Realism, and Aesthetics

TL;DR

This paper introduces a novel feature supervision paradigm for Multimodal Diffusion Transformers that enhances portrait generation by improving alignment, realism, and aesthetics simultaneously without additional inference costs.

Contribution

It proposes a lightweight cross-modal alignment mechanism and aesthetic signal mining to overcome the trade-offs in text-to-image portrait generation.

Findings

Improves text-image alignment, realism, and aesthetics simultaneously.

Maintains original model generalization without overfitting.

Achieves state-of-the-art results on MM-DiT benchmarks.

Abstract

Text-to-image diffusion models often face a severe trilemma in human portrait generation: text-image alignment, photorealism, and human-perceived aesthetics inherently inhibit one another. Supervised Fine-Tuning (SFT) is an effective method for enhancing the photorealism of image generation. However, it often leads to overfitting to the training dataset, corrupts pre-trained image priors, and degrades alignment or aesthetics. To break this bottleneck, we propose a feature supervision paradigm for Multimodal Diffusion Transformers (MM-DiT). Specifically, we introduce a lightweight cross-modal alignment mechanism that implicitly extracts multi-granularity vision-aligned text representations from SigLIP 2 and applies supervision to the image branch of MM-DiT during the training stage, with zero extra inference overhead. Our method injects vision-aligned text guidance while preserving the base model's original generalization, avoiding degradation caused by SFT. Furthermore, our method directly mines implicit multi-granularity aesthetic signals from pre-trained vision foundation models to optimize human-perceived aesthetics. Extensive experiments on MM-DiTs show that our method pushes the Pareto frontier and achieves synergistic improvements across text-image alignment, photorealism, and human-perceived aesthetics.