RAGDiffusion: Faithful Cloth Generation via External Knowledge Assimilation

TL;DR

RAGDiffusion is a novel framework that combines retrieval-augmented generation with diffusion models to produce highly faithful clothing images, addressing structural hallucinations and texture distortions in complex scenarios.

Contribution

It introduces a retrieval-based structure aggregation and omni-level texture alignment, pioneering high-fidelity garment generation with external knowledge integration.

Findings

Significant improvement in structural accuracy and texture fidelity.

Effective mitigation of hallucinations in complex clothing scenarios.

Outperforms existing models on real-world datasets.

Abstract

Standard clothing asset generation involves restoring forward-facing flat-lay garment images displayed on a clear background by extracting clothing information from diverse real-world contexts, which presents significant challenges due to highly standardized structure sampling distributions and clothing semantic absence in complex scenarios. Existing models have limited spatial perception, often exhibiting structural hallucinations and texture distortion in this high-specification generative task. To address this issue, we propose a novel Retrieval-Augmented Generation (RAG) framework, termed RAGDiffusion, to enhance structure determinacy and mitigate hallucinations by assimilating knowledge from language models and external databases. RAGDiffusion consists of two processes: (1) Retrieval-based structure aggregation, which employs contrastive learning and a Structure Locally Linear Embedding (SLLE) to derive global structure and spatial landmarks, providing both soft and hard guidance to counteract structural ambiguities; and (2) Omni-level faithful garment generation, which introduces a coarse-to-fine texture alignment that ensures fidelity in pattern and detail components within the diffusing. Extensive experiments on challenging real-world datasets demonstrate that RAGDiffusion synthesizes structurally and texture-faithful clothing assets with significant performance improvements, representing a pioneering effort in high-specification faithful generation with RAG to confront intrinsic hallucinations and enhance fidelity.