Measuring Diversity in Synthetic Datasets

TL;DR

This paper introduces DCScore, a new classification-based method for measuring the diversity of synthetic datasets in NLP, which is more effective and computationally efficient than existing approaches.

Contribution

The paper presents DCScore, a novel diversity measurement method that is theoretically grounded, correlates well with diversity truths, and reduces computational costs.

Findings

DCScore correlates strongly with diversity pseudo-truths

It reduces computational costs compared to existing methods

Theoretical verification confirms its validity as a diversity measure

Abstract

Large language models (LLMs) are widely adopted to generate synthetic datasets for various natural language processing (NLP) tasks, such as text classification and summarization. However, accurately measuring the diversity of these synthetic datasets-an aspect crucial for robust model performance-remains a significant challenge. In this paper, we introduce DCScore, a novel method for measuring synthetic dataset diversity from a classification perspective. Specifically, DCScore formulates diversity evaluation as a sample classification task, leveraging mutual relationships among samples. We further provide theoretical verification of the diversity-related axioms satisfied by DCScore, highlighting its role as a principled diversity evaluation method. Experimental results on synthetic datasets reveal that DCScore enjoys a stronger correlation with multiple diversity pseudo-truths of evaluated datasets, underscoring its effectiveness. Moreover, both empirical and theoretical evidence demonstrate that DCScore substantially reduces computational costs compared to existing methods. Code is available at: https://github.com/bluewhalelab/dcscore.