UniHetero: Could Generation Enhance Understanding for Vision-Language-Model at Large Data Scale?

TL;DR

This paper investigates whether large-scale vision-language models can enhance understanding through generation, finding that semantic-level generation improves understanding and reveals better data scaling, while pixel-level objectives may hinder performance.

Contribution

The study introduces UniHetero, a unified model demonstrating that semantic generation at large scale enhances understanding and data utilization, with effective autoregression on input embeddings.

Findings

Semantic generation improves understanding at large scale.

Pixel-level objectives can degrade understanding performance.

Autoregression on input embeddings captures visual details effectively.

Abstract

Vision-language large models are moving toward the unification of visual understanding and visual generation tasks. However, whether generation can enhance understanding is still under-explored on large data scale. In this work, we analysis the unified structure with a concise model, UniHetero, under large-scale pretraining (>200M samples). Our key observations are: (1) Generation can improve understanding, but Only if you generate Semantics, Not Pixels. A common assumption in unified vision-language models is that adding generation will naturally strengthen understanding. However, this is not always true at scale. At 200M+ pretraining samples, generation helps understanding only when it operates at the semantic level, i.e. when the model learns to autoregress high-level visual representations inside the LLM. Once pixel-level objectives (e.g., diffusion losses) directly interfere with the LLM, understanding performance often degrades. (2) Generation reveals a superior Data Scaling trend and higher Data Utilization. Unified generation-understanding demonstrates a superior scaling trend compared to understanding alone, revealing a more effective way to learn vision-only knowledge directive from vision modality rather than captioning to text. (3) Autoregression on Input Embedding is effective to capture visual details. Compared to the commonly-used vision encoder, make visual autoregression on input embedding shows less cumulative error and is modality independent, which can be extend to all modalities. The learned semantic representations capture visual information such as objects, locations, shapes, and colors; further enable pixel-level image generation.