Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collections

TL;DR

This paper introduces new quantitative metrics for assessing text collections based on diversity, density, and homogeneity, which correlate with classification performance and enhance understanding of textual data characteristics.

Contribution

It proposes novel metrics for measuring dispersion, sparsity, and uniformity in text collections, validated through simulations and real-world experiments.

Findings

Metrics align with human intuition

Strong correlation with BERT classification performance

Potential to guide dataset analysis and model development

Abstract

Summarizing data samples by quantitative measures has a long history, with descriptive statistics being a case in point. However, as natural language processing methods flourish, there are still insufficient characteristic metrics to describe a collection of texts in terms of the words, sentences, or paragraphs they comprise. In this work, we propose metrics of diversity, density, and homogeneity that quantitatively measure the dispersion, sparsity, and uniformity of a text collection. We conduct a series of simulations to verify that each metric holds desired properties and resonates with human intuitions. Experiments on real-world datasets demonstrate that the proposed characteristic metrics are highly correlated with text classification performance of a renowned model, BERT, which could inspire future applications.