ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations

TL;DR

ECLIPSE introduces a resource-efficient contrastive learning method for text-to-image priors, achieving high performance with significantly fewer parameters and less data, thus reducing computational costs in image generation tasks.

Contribution

ECLIPSE is a novel contrastive learning approach that distills knowledge from pre-trained vision-language models into a compact T2I prior, outperforming larger models in resource-limited settings.

Findings

ECLIPSE trained prior surpasses baseline T2I priors in preference scores.

Achieves comparable performance to state-of-the-art large models.

Reduces resource requirements by over 96% in parameters and data.

Abstract

Text-to-image (T2I) diffusion models, notably the unCLIP models (e.g., DALL-E-2), achieve state-of-the-art (SOTA) performance on various compositional T2I benchmarks, at the cost of significant computational resources. The unCLIP stack comprises T2I prior and diffusion image decoder. The T2I prior model alone adds a billion parameters compared to the Latent Diffusion Models, which increases the computational and high-quality data requirements. We introduce ECLIPSE, a novel contrastive learning method that is both parameter and data-efficient. ECLIPSE leverages pre-trained vision-language models (e.g., CLIP) to distill the knowledge into the prior model. We demonstrate that the ECLIPSE trained prior, with only 3.3% of the parameters and trained on a mere 2.8% of the data, surpasses the baseline T2I priors with an average of 71.6% preference score under resource-limited setting. It also attains performance on par with SOTA big models, achieving an average of 63.36% preference score in terms of the ability to follow the text compositions. Extensive experiments on two unCLIP diffusion image decoders, Karlo and Kandinsky, affirm that ECLIPSE priors consistently deliver high performance while significantly reducing resource dependency.