OmniThoughtVis: A Scalable Distillation Pipeline for Deployable Multimodal Reasoning Models

TL;DR

OmniThoughtVis introduces a scalable pipeline for creating high-quality multimodal reasoning data and distilling large models into smaller, efficient models with improved reasoning capabilities for real-world deployment.

Contribution

The paper presents a novel data curation and distillation pipeline that transfers reasoning skills from large models to smaller ones, enhancing their performance on multimodal tasks.

Findings

Distilled models outperform baseline models on multiple benchmarks.

The pipeline creates a high-quality dataset of 1.8 million samples.

Smaller models achieve comparable or better performance than larger models.

Abstract

Recent multimodal large language models (MLLMs) have shown strong chain-of-thought (CoT) reasoning ability on vision-language tasks, but their direct deployment in real-world systems is often limited by latency and resource constraints. In practice, smaller MLLMs are preferred for online serving, yet their reasoning performance is bottlenecked by the lack of large-scale, high-quality multimodal CoT supervision. In this paper, we present OmniThoughtVis, a scalable data curation and distillation pipeline for transferring multimodal reasoning capabilities from high-capacity teacher models to smaller, deployment-oriented MLLMs. Starting from a diverse open-source seed pool, our pipeline generates structured CoT traces and performs joint annotation of reasoning difficulty, answer quality, and semantic task tags. To maintain data quality at scale, we combine rule-based filtering, difficulty-aware selection, and tag-based diversity sampling, resulting in a curated corpus of 1.8M samples that supports controllable subset construction for downstream training. We use OmniThoughtVis to distill Qwen3-VL models from 2B to 8B parameters and evaluate them on nine multimodal reasoning benchmarks. The resulting distilled models show consistent gains across model scales, including improvements of up to +16.8 points on MathVerse and +5.6 points on MMMU-Pro for the 4B model. Notably, the distilled 4B model matches or surpasses the undistilled 8B baseline on several tasks, highlighting the practical value of scalable reasoning distillation for deployment-oriented MLLMs.