SkillRater: Untangling Capabilities in Multimodal Data

TL;DR

SkillRater introduces a multidimensional data filtering framework that trains specialized raters for different capabilities, improving model performance across vision language tasks by preserving diverse high-quality samples.

Contribution

The paper proposes a novel multi-capability data filtering method using meta-learned specialized raters, outperforming traditional scalar scoring approaches.

Findings

Improves vision language model performance by up to 5.63% on key capabilities.

Demonstrates that learned rater signals are near orthogonal, indicating independent quality dimensions.

Outperforms unfiltered training and monolithic filtering baselines.

Abstract

Data curation methods typically assign samples a single quality score. We argue this scalar framing is fundamentally limited: when training requires multiple distinct capabilities, a monolithic scorer cannot maximize useful signals for all of them simultaneously. Quality is better understood as multidimensional, with each dimension corresponding to a capability the model must acquire. We introduce SkillRater, a framework that decomposes data filtering into specialized raters - one per capability, each trained via meta-learning on a disjoint validation objective - and composes their scores through a progressive selection rule: at each training stage, a sample is retained if any rater ranks it above a threshold that tightens over time, preserving diversity early while concentrating on high-value samples late. We validate this approach on vision language models, decomposing quality into three capability dimensions: visual understanding, OCR, and STEM reasoning. At 2B parameters, SkillRater improves over unfiltered baselines by 5.63% on visual understanding, 2.00% on OCR, and 3.53% on STEM on held out benchmarks. The learned rater signals are near orthogonal, confirming that the decomposition captures genuinely independent quality dimensions and explaining why it outperforms both unfiltered training and monolithic learned filtering.