MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modeling

TL;DR

MMAR introduces a lossless, continuous-valued multi-modal auto-regressive framework that improves image understanding and generation, surpassing existing models in accuracy and quality while maintaining scalability.

Contribution

The paper proposes a novel MMAR framework that avoids information loss by using continuous image tokens and disentangles diffusion from auto-regressive modeling, with proven training techniques.

Findings

Outperforms existing multi-modal models on 18 benchmarks

Generates high-quality images with improved understanding capabilities

Scalable with larger datasets and model sizes

Abstract

Recent advancements in multi-modal large language models have propelled the development of joint probabilistic models capable of both image understanding and generation. However, we have identified that recent methods suffer from loss of image information during understanding task, due to either image discretization or diffusion denoising steps. To address this issue, we propose a novel Multi-Modal Auto-Regressive (MMAR) probabilistic modeling framework. Unlike discretization line of method, MMAR takes in continuous-valued image tokens to avoid information loss in an efficient way. Differing from diffusion-based approaches, we disentangle the diffusion process from auto-regressive backbone model by employing a light-weight diffusion head on top each auto-regressed image patch embedding. In this way, when the model transits from image generation to understanding through text generation, the backbone model's hidden representation of the image is not limited to the last denoising step. To successfully train our method, we also propose a theoretically proven technique that addresses the numerical stability issue and a training strategy that balances the generation and understanding task goals. Extensive evaluations on 18 image understanding benchmarks show that MMAR significantly outperforms most of the existing joint multi-modal models, surpassing the method that employs pre-trained CLIP vision encoder. Meanwhile, MMAR is able to generate high quality images. We also show that our method is scalable with larger data and model size.