GRIP: Geometric Refinement and Adaptive Information Potential for Data Efficiency

TL;DR

GRIP introduces a geometric framework for data-efficient training of large language models, dynamically allocating sampling based on semantic cluster information to improve performance on large-scale tasks.

Contribution

It unifies global distribution balancing with local instance selection through a geometric space model and adaptive sampling strategies, enhancing data efficiency.

Findings

Outperforms state-of-the-art baselines on MoE models up to 300B tokens.

Surpasses the performance of models trained on three times larger uncurated datasets.

Establishes a geometric foundation for adaptive data curation in large-scale pre-training.

Abstract

The performance of Large Language Models (LLMs) is increasingly governed by data efficiency rather than raw scaling volume. However, existing selection methods often decouple global distribution balancing from local instance selection, compromising the hierarchical integrity of the training set. We introduce \textbf{GRIP} (Geometric Refinement and Adaptive Information Potential), a framework that unifies these dimensions by modeling the corpus as an information-dense geometric space. GRIP employs a \textbf{Rapid Adaptation Probe (RAP)} to quantify the information potential of semantic clusters, dynamically re-allocating the sampling budget to regions with the highest representation deficits. Subsequently, we perform Intra-Cluster Selection using a \textbf{length-rectified geometric prior} to counteract embedding density artifacts and preserve long-tail logical sequences. Extensive evaluations on Mixture-of-Experts (MoE) models up to 300B tokens demonstrate that GRIP consistently outperforms state-of-the-art baselines, \textbf{surpassing the performance of models trained on $3\times$ larger uncurated datasets}. Our work establishes a robust geometric foundation for adaptive data curation in large-scale pre-training.