Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generation

TL;DR

Open-MAGVIT2 is an open-source project that replicates Google's large-vocabulary tokenizer, achieving state-of-the-art image reconstruction and enabling scalable auto-regressive visual generation with improved quality.

Contribution

It introduces an open-source large-vocabulary tokenizer, explores its application in scalable auto-regressive models, and proposes novel token factorization techniques for better image generation.

Findings

Achieves state-of-the-art reconstruction on ImageNet and UCF.

Outperforms Cosmos in zero-shot benchmarks.

Provides scalable auto-regressive models from 300M to 1.5B parameters.

Abstract

The Open-MAGVIT2 project produces an open-source replication of Google's MAGVIT-v2 tokenizer, a tokenizer with a super-large codebook (i.e., $2^{18}$ codes), and achieves the state-of-the-art reconstruction performance on ImageNet and UCF benchmarks. We also provide a tokenizer pre-trained on large-scale data, significantly outperforming Cosmos on zero-shot benchmarks (1.93 vs. 0.78 rFID on ImageNet original resolution). Furthermore, we explore its application in plain auto-regressive models to validate scalability properties, producing a family of auto-regressive image generation models ranging from 300M to 1.5B. To assist auto-regressive models in predicting with a super-large vocabulary, we factorize it into two sub-vocabulary of different sizes by asymmetric token factorization, and further introduce ``next sub-token prediction'' to enhance sub-token interaction for better generation quality. We release all models and codes to foster innovation and creativity in the field of auto-regressive visual generation.