Kelix Technical Report

TL;DR

Kelix is a fully discrete autoregressive model that enhances multimodal understanding by bridging the gap between discrete visual tokens and continuous representations, enabling more effective unified language and vision processing.

Contribution

Kelix introduces a novel fully discrete autoregressive model that improves multimodal understanding, addressing limitations of previous discrete visual tokenization methods.

Findings

Kelix achieves comparable understanding to continuous-feature VLMs.

Discrete visual tokens in Kelix retain more information than previous methods.

Kelix demonstrates improved performance in multimodal tasks.

Abstract

Autoregressive large language models (LLMs) scale well by expressing diverse tasks as sequences of discrete natural-language tokens and training with next-token prediction, which unifies comprehension and generation under self-supervision. Extending this paradigm to multimodal data requires a shared, discrete representation across modalities. However, most vision-language models (VLMs) still rely on a hybrid interface: discrete text tokens paired with continuous Vision Transformer (ViT) features. Because supervision is largely text-driven, these models are often biased toward understanding and cannot fully leverage large-scale self-supervised learning on non-text data. Recent work has explored discrete visual tokenization to enable fully autoregressive multimodal modeling, showing promising progress toward unified understanding and generation. Yet existing discrete vision tokens frequently lose information due to limited code capacity, resulting in noticeably weaker understanding than continuous-feature VLMs. We present Kelix, a fully discrete autoregressive unified model that closes the understanding gap between discrete and continuous visual representations.