VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modeling

TL;DR

VAR-CLIP introduces a novel text-to-image generation approach that combines visual auto-regressive modeling with CLIP, enabling high-quality, faithful, and aesthetic image synthesis guided by textual captions.

Contribution

It integrates visual auto-regressive modeling with CLIP for flexible text-guided image generation and constructs a large dataset for training on extensive datasets like ImageNet.

Findings

High fidelity and aesthetic quality in generated images

Effective caption guidance through word positioning analysis

Successful training on large-scale datasets like ImageNet

Abstract

VAR is a new generation paradigm that employs 'next-scale prediction' as opposed to 'next-token prediction'. This innovative transformation enables auto-regressive (AR) transformers to rapidly learn visual distributions and achieve robust generalization. However, the original VAR model is constrained to class-conditioned synthesis, relying solely on textual captions for guidance. In this paper, we introduce VAR-CLIP, a novel text-to-image model that integrates Visual Auto-Regressive techniques with the capabilities of CLIP. The VAR-CLIP framework encodes captions into text embeddings, which are then utilized as textual conditions for image generation. To facilitate training on extensive datasets, such as ImageNet, we have constructed a substantial image-text dataset leveraging BLIP2. Furthermore, we delve into the significance of word positioning within CLIP for the purpose of caption guidance. Extensive experiments confirm VAR-CLIP's proficiency in generating fantasy images with high fidelity, textual congruence, and aesthetic excellence. Our project page are https://github.com/daixiangzi/VAR-CLIP