Sketch-in-Latents: Eliciting Unified Reasoning in MLLMs

TL;DR

SkiLa introduces a unified reasoning framework for multimodal models that enables them to generate and incorporate continuous visual embeddings during multi-step reasoning, improving performance on vision-centric tasks.

Contribution

It presents SkiLa, a novel paradigm allowing MLLMs to natively generate latent visual tokens, unifying visual and textual reasoning without predefined toolkits.

Findings

Outperforms existing models on vision-centric tasks

Demonstrates strong generalization across multi-modal benchmarks

Enables seamless integration of visual imagination in reasoning processes

Abstract

While Multimodal Large Language Models (MLLMs) excel at visual understanding tasks through text reasoning, they often fall short in scenarios requiring visual imagination. Unlike current works that take predefined external toolkits or generate images during thinking, however, humans can form flexible visual-text imagination and interactions during thinking without predefined toolkits, where one important reason is that humans construct the visual-text thinking process in a unified space inside the brain. Inspired by this capability, given that current MLLMs already encode visual and text information in the same feature space, we hold that visual tokens can be seamlessly inserted into the reasoning process carried by text tokens, where ideally, all visual imagination processes can be encoded by the latent features. To achieve this goal, we propose Sketch-in-Latents (SkiLa), a novel paradigm for unified multi-modal reasoning that expands the auto-regressive capabilities of MLLMs to natively generate continuous visual embeddings, termed latent sketch tokens, as visual thoughts. During multi-step reasoning, the model dynamically alternates between textual thinking mode for generating textual think tokens and visual sketching mode for generating latent sketch tokens. A latent visual semantics reconstruction mechanism is proposed to ensure these latent sketch tokens are semantically grounded. Extensive experiments demonstrate that SkiLa achieves superior performance on vision-centric tasks while exhibiting strong generalization to diverse general multi-modal benchmarks. Codes will be released at https://github.com/TungChintao/SkiLa.