LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?

TL;DR

LatentRouter is a novel routing method for multimodal large language models that predicts model utility based on input features, enabling dynamic selection tailored to multimodal task requirements.

Contribution

It introduces a counterfactual utility prediction approach with latent communication for improved model routing in multimodal tasks.

Findings

LatentRouter outperforms fixed-model and baseline routers on benchmark datasets.

Gains are most significant on tasks requiring visual, layout-sensitive, or reasoning skills.

Latent communication between model states is key to the improved performance.

Abstract

Multimodal large language models (MLLMs) have heterogeneous strengths across OCR, chart understanding, spatial reasoning, visual question answering, cost, and latency. Effective MLLM routing therefore requires more than estimating query difficulty: a router must match the multimodal requirements of the current image-question input with the capabilities of each candidate model. We propose LatentRouter, a router that formulates MLLM routing as counterfactual multimodal utility prediction. Given an image-question query, LatentRouter extracts learned multimodal routing capsules, represents each candidate MLLM with a model capability token, and performs latent communication between these states to estimate how each model would perform if selected. A distributional outcome head predicts model-specific counterfactual quality, while a bounded capsule correction refines close decisions without allowing residual signals to dominate the prediction. The resulting utility-based policy supports performance-oriented and performance-cost routing, and handles changing candidate pools through shared per-model scoring with availability masking. Experiments on MMR-Bench and VL-RouterBench show that LatentRouter outperforms fixed-model, feature-level, and learned-router baselines. Additional analyses show that the gains are strongest on multimodal task groups where model choice depends on visual, layout-sensitive, or reasoning-oriented requirements, and that latent communication is the main contributor to the improvement. The code is available at: https://github.com/LabRAI/LatentRouter.