What Matters in Data Curation for Multimodal Reasoning? Insights from the DCVLR Challenge

TL;DR

This paper analyzes data curation strategies for multimodal reasoning, emphasizing the importance of example difficulty and alignment over dataset size or diversity, based on insights from the DCVLR challenge.

Contribution

It demonstrates that difficulty-based example selection significantly improves performance in multimodal reasoning datasets, challenging the emphasis on dataset size and diversity.

Findings

Difficulty-based selection drives performance gains.

Increasing dataset size mainly reduces variance.

Diversity and synthetic augmentation often degrade performance.

Abstract

We study data curation for multimodal reasoning through the NeurIPS 2025 Data Curation for Vision-Language Reasoning (DCVLR) challenge, which isolates dataset selection by fixing the model and training protocol. Using a compact curated dataset derived primarily from Walton Multimodal Cold Start, our submission placed first in the challenge. Through post-competition ablations, we show that difficulty-based example selection on an aligned base dataset is the dominant driver of performance gains. Increasing dataset size does not reliably improve mean accuracy under the fixed training recipe, but mainly reduces run-to-run variance, while commonly used diversity and synthetic augmentation heuristics provide no additional benefit and often degrade performance. These results characterize DCVLR as a saturation-regime evaluation and highlight the central role of alignment and difficulty in data-efficient multimodal reasoning.