Enhancing Diffusion Face Generation with Contrastive Embeddings and SegFormer Guidance

TL;DR

This paper benchmarks diffusion models for face generation, introducing contrastive embeddings and SegFormer guidance to improve semantic control and quality in limited data scenarios.

Contribution

It integrates InfoNCE contrastive loss and SegFormer segmentation encoder into diffusion models, enhancing attribute control and semantic alignment.

Findings

Contrastive embeddings improve attribute controllability.

SegFormer guidance enhances semantic alignment.

Benchmark results demonstrate improved face generation quality.

Abstract

We present a benchmark of diffusion models for human face generation on a small-scale CelebAMask-HQ dataset, evaluating both unconditional and conditional pipelines. Our study compares UNet and DiT architectures for unconditional generation and explores LoRA-based fine-tuning of pretrained Stable Diffusion models as a separate experiment. Building on the multi-conditioning approach of Giambi and Lisanti, which uses both attribute vectors and segmentation masks, our main contribution is the integration of an InfoNCE loss for attribute embedding and the adoption of a SegFormer-based segmentation encoder. These enhancements improve the semantic alignment and controllability of attribute-guided synthesis. Our results highlight the effectiveness of contrastive embedding learning and advanced segmentation encoding for controlled face generation in limited data settings.