Multimodal Latent Reasoning via Predictive Embeddings

TL;DR

Pearl introduces a latent space reasoning framework for visual language models that learns from tool-use trajectories without explicit tool invocation, improving efficiency and multi-step reasoning capabilities.

Contribution

The paper presents Pearl, a novel latent space predictive embedding method that eliminates the need for explicit tool calls and supports multi-step reasoning in multimodal tasks.

Findings

Pearl matches or outperforms supervised fine-tuning and reconstruction-based methods.

Reconstruction-based methods mainly learn embeddings, not image edits, in latent space.

Pearl supports multiple tool calls in reasoning trajectories.

Abstract

Tool-augmented multimodal reasoning enables visual language models (VLMs) to improve perception by interacting with external tools (e.g., cropping, depth estimation). However, such approaches incur substantial inference overhead, require specialized supervision, and are prone to erroneous tool calls. We propose Pearl (Predictive Embedding Alignment for Reasoning in Latent space), a JEPA-inspired framework that learns from expert tool-use trajectories entirely in the latent space, eliminating the need for explicit tool invocation at inference time. Unlike reconstruction-based latent reasoning methods, which autoregressively generate latent tokens and suffer from training-inference mismatch and limited support for multi-step tool use, Pearl directly learns predictive embeddings from multimodal trajectories while preserving the standard vision-language generation pipeline: it is model-agnostic, simple to train, and naturally supports trajectories with multiple tool calls. Experiments across multiple perception benchmarks show that Pearl matches or outperforms standard supervised fine-tuning and reconstruction-based latent reasoning approaches. Furthermore, we provide empirical evidence that reconstruction-based methods primarily learn embeddings rather than image edits in latent space, motivating predictive embedding learning as a more principled alternative.