Closing the Data Loop: Using OpenDataArena to Engineer Superior Training Datasets

TL;DR

This paper introduces a systematic, feedback-driven framework called OpenDataArena for constructing superior training datasets for large language models, leading to state-of-the-art results and improved data efficiency.

Contribution

It presents a novel closed-loop dataset engineering paradigm using value-based rankings and multi-dimensional analysis, with new datasets demonstrating significant performance improvements.

Findings

State-of-the-art results on math benchmarks with ODA-Math-460k

Superior multi-domain instruction datasets outperform larger baselines

Enhanced data efficiency and model reasoning capabilities

Abstract

The construction of Supervised Fine-Tuning (SFT) datasets is a critical yet under-theorized stage in the post-training of Large Language Models (LLMs), as prevalent practices often rely on heuristic aggregation without a systematic understanding of how individual samples contribute to model performance. In this report, we propose a paradigm shift from ad-hoc curation to a closed-loop dataset engineering framework using OpenDataArena (ODA), which leverages value-anchored rankings and multi-dimensional analysis to transform value benchmarking into feedback signals guiding dataset construction. We instantiate this methodology through two new datasets: \textbf{ODA-Math-460k}, a specialized mathematics reasoning dataset that utilizes a novel two-stage difficulty-aware pipeline to achieve State-of-the-Art (SOTA) results on benchmarks such as AIME and HMMT, and \textbf{ODA-Mixture (100k \& 500k)}, a series of multi-domain instruction datasets built via an ``Anchor-and-Patch'' strategy that outperforms significantly larger open-source baselines. Our empirical results demonstrate that ODA-driven datasets significantly improve both domain-specific reasoning and general utility while achieving superior data efficiency, validating a transition toward data-centric AI where transparent evaluation serves as the primary engine for engineering high-quality training data.