NativeTok: Native Visual Tokenization for Improved Image Generation

TL;DR

NativeTok introduces a novel visual tokenization method that enforces causal dependencies, leading to improved image generation coherence by embedding relational constraints directly into token sequences.

Contribution

It proposes native visual tokenization with causal dependencies, and a new framework NativeTok combining MIT and MoCET for efficient, constrained image tokenization and generation.

Findings

Enhanced image reconstruction quality.

Better coherence in generated images.

Efficient training with Hierarchical Native Training.

Abstract

VQ-based image generation typically follows a two-stage pipeline: a tokenizer encodes images into discrete tokens, and a generative model learns their dependencies for reconstruction. However, improved tokenization in the first stage does not necessarily enhance the second-stage generation, as existing methods fail to constrain token dependencies. This mismatch forces the generative model to learn from unordered distributions, leading to bias and weak coherence. To address this, we propose native visual tokenization, which enforces causal dependencies during tokenization. Building on this idea, we introduce NativeTok, a framework that achieves efficient reconstruction while embedding relational constraints within token sequences. NativeTok consists of: (1) a Meta Image Transformer (MIT) for latent image modeling, and (2) a Mixture of Causal Expert Transformer (MoCET), where each lightweight expert block generates a single token conditioned on prior tokens and latent features. We further design a Hierarchical Native Training strategy that updates only new expert blocks, ensuring training efficiency. Extensive experiments demonstrate the effectiveness of NativeTok.