GarmentAligner: Text-to-Garment Generation via Retrieval-augmented Multi-level Corrections

TL;DR

GarmentAligner is a novel diffusion model that improves text-to-garment image generation by using retrieval-augmented multi-level corrections for fine-grained semantic and component alignment.

Contribution

It introduces an automatic component extraction pipeline and multi-level correction losses to enhance semantic, spatial, and quantitative alignment in garment generation.

Findings

Achieves superior fidelity compared to existing models.

Enhances fine-grained semantic alignment of garment components.

Utilizes retrieval augmentation and contrastive learning for better perception.

Abstract

General text-to-image models bring revolutionary innovation to the fields of arts, design, and media. However, when applied to garment generation, even the state-of-the-art text-to-image models suffer from fine-grained semantic misalignment, particularly concerning the quantity, position, and interrelations of garment components. Addressing this, we propose GarmentAligner, a text-to-garment diffusion model trained with retrieval-augmented multi-level corrections. To achieve semantic alignment at the component level, we introduce an automatic component extraction pipeline to obtain spatial and quantitative information of garment components from corresponding images and captions. Subsequently, to exploit component relationships within the garment images, we construct retrieval subsets for each garment by retrieval augmentation based on component-level similarity ranking and conduct contrastive learning to enhance the model perception of components from positive and negative samples. To further enhance the alignment of components across semantic, spatial, and quantitative granularities, we propose the utilization of multi-level correction losses that leverage detailed component information. The experimental findings demonstrate that GarmentAligner achieves superior fidelity and fine-grained semantic alignment when compared to existing competitors.