CustomContrast: A Multilevel Contrastive Perspective For Subject-Driven Text-to-Image Customization

TL;DR

CustomContrast introduces a multilevel contrastive learning framework with a multimodal encoder to improve subject-driven text-to-image customization by better decoupling intrinsic subject attributes from irrelevant features.

Contribution

The paper proposes a novel multilevel contrastive learning approach with a multimodal feature injection encoder for enhanced subject customization in text-to-image generation.

Findings

Improves subject similarity in generated images.

Enhances text controllability for subject customization.

Effective in decoupling intrinsic and irrelevant attributes.

Abstract

Subject-driven text-to-image (T2I) customization has drawn significant interest in academia and industry. This task enables pre-trained models to generate novel images based on unique subjects. Existing studies adopt a self-reconstructive perspective, focusing on capturing all details of a single image, which will misconstrue the specific image's irrelevant attributes (e.g., view, pose, and background) as the subject intrinsic attributes. This misconstruction leads to both overfitting or underfitting of irrelevant and intrinsic attributes of the subject, i.e., these attributes are over-represented or under-represented simultaneously, causing a trade-off between similarity and controllability. In this study, we argue an ideal subject representation can be achieved by a cross-differential perspective, i.e., decoupling subject intrinsic attributes from irrelevant attributes via contrastive learning, which allows the model to focus more on intrinsic attributes through intra-consistency (features of the same subject are spatially closer) and inter-distinctiveness (features of different subjects have distinguished differences). Specifically, we propose CustomContrast, a novel framework, which includes a Multilevel Contrastive Learning (MCL) paradigm and a Multimodal Feature Injection (MFI) Encoder. The MCL paradigm is used to extract intrinsic features of subjects from high-level semantics to low-level appearance through crossmodal semantic contrastive learning and multiscale appearance contrastive learning. To facilitate contrastive learning, we introduce the MFI encoder to capture cross-modal representations. Extensive experiments show the effectiveness of CustomContrast in subject similarity and text controllability.