D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery

TL;DR

D3-Gym is a new dataset of 565 real-world scientific tasks with verifiable environments, designed to advance data-driven discovery by providing executable environments, reference solutions, and evaluation scripts.

Contribution

It introduces the first automatically constructed dataset with verifiable environments for scientific discovery, including a comprehensive evaluation of its verification signals.

Findings

Evaluation scripts achieve 87.5% agreement with human standards.

Training on D3-Gym improves model performance on ScienceAgentBench.

D3-Gym significantly reduces the gap with proprietary models.

Abstract

Despite recent progress in language models and agents for scientific data-driven discovery, further advancing their capabilities is held back by the absence of verifiable environments representing real-world scientific tasks. To fill this gap, we introduce D3-Gym, the first automatically constructed dataset with verifiable environments for scientific Data-Driven Discovery. D3-Gym comprises (1) 565 tasks sourced from 239 real scientific repositories across four disciplines where (2) each task is equipped with a natural language instruction, an executable environment with pre-installed dependencies, input dataset and artifact previews, a reference code solution, and an automatically synthesized evaluation script. Rigorous evaluation of the quality of the verification signal in D3-Gym confirms that our evaluation scripts achieve 87.5% agreement with human-annotated gold standards and strong alignment in domain-specific evaluation logic, showing their scientific soundness. Further, training on trajectories sampled from D3-Gym yields consistent and substantial gains across Qwen3 models of varying sizes on ScienceAgentBench, boosting Qwen3-32B by 7.8 absolute points and substantially shrinking the gap with strong proprietary models. All D3-Gym artifacts (environments, creation workflow, trajectories, and models) can be found at https://github.com/OSU-NLP-Group/D3-Gym.