CogView: Mastering Text-to-Image Generation via Transformers

TL;DR

CogView is a large Transformer-based model that significantly advances text-to-image generation by integrating VQ-VAE tokenization, demonstrating superior performance and versatile fine-tuning for various applications.

Contribution

Introduces CogView, a 4-billion-parameter Transformer with VQ-VAE, achieving state-of-the-art results and offering new fine-tuning strategies for diverse downstream tasks.

Findings

Achieves state-of-the-art FID on MS COCO dataset

Outperforms previous GAN-based models and DALL-E

Demonstrates effective fine-tuning for multiple applications

Abstract

Text-to-Image generation in the general domain has long been an open problem, which requires both a powerful generative model and cross-modal understanding. We propose CogView, a 4-billion-parameter Transformer with VQ-VAE tokenizer to advance this problem. We also demonstrate the finetuning strategies for various downstream tasks, e.g. style learning, super-resolution, text-image ranking and fashion design, and methods to stabilize pretraining, e.g. eliminating NaN losses. CogView achieves the state-of-the-art FID on the blurred MS COCO dataset, outperforming previous GAN-based models and a recent similar work DALL-E.