SCAR: A Characterization Scheme for Multi-Modal Dataset

TL;DR

SCAR is a novel framework for characterizing datasets' intrinsic structural properties across scale, coverage, authenticity, and richness, enabling better understanding and efficient augmentation of multi-modal datasets for foundation models.

Contribution

Introduces SCAR, a robust scheme for dataset characterization that remains invariant under scaling and guides data augmentation for improved model generalization.

Findings

SCAR effectively predicts data utility across diverse datasets.

Foundation data can preserve model generalization without retraining.

SCAR-guided data expansion improves multimodal dataset quality.

Abstract

Foundation models exhibit remarkable generalization across diverse tasks, largely driven by the characteristics of their training data. Recent data-centric methods like pruning and compression aim to optimize training but offer limited theoretical insight into how data properties affect generalization, especially the data characteristics in sample scaling. Traditional perspectives further constrain progress by focusing predominantly on data quantity and training efficiency, often overlooking structural aspects of data quality. In this study, we introduce SCAR, a principled scheme for characterizing the intrinsic structural properties of datasets across four key measures: Scale, Coverage, Authenticity, and Richness. Unlike prior data-centric measures, SCAR captures stable characteristics that remain invariant under dataset scaling, providing a robust and general foundation for data understanding. Leveraging these structural properties, we introduce Foundation Data-a minimal subset that preserves the generalization behavior of the full dataset without requiring model-specific retraining. We model single-modality tasks as step functions and estimate the distribution of the foundation data size to capture step-wise generalization bias across modalities in the target multi-modal dataset. Finally, we develop a SCAR-guided data completion strategy based on this generalization bias, which enables efficient, modality-aware expansion of modality-specific characteristics in multimodal datasets. Experiments across diverse multi-modal datasets and model architectures validate the effectiveness of SCAR in predicting data utility and guiding data acquisition. Code is available at https://github.com/McAloma/SCAR.