Measuring the Measuring Tools: An Automatic Evaluation of Semantic Metrics for Text Corpora

TL;DR

This paper introduces automatic, interpretable evaluation measures for semantic similarity metrics at the corpus level, enabling better comparison and understanding of their behavior in NLP applications.

Contribution

It proposes a novel set of evaluation measures for semantic similarity metrics, facilitating meaningful comparison and analysis of their characteristics.

Findings

New metrics better identify semantic distributional mismatch

Classical metrics are more sensitive to surface text perturbations

Evaluation measures effectively capture fundamental metric characteristics

Abstract

The ability to compare the semantic similarity between text corpora is important in a variety of natural language processing applications. However, standard methods for evaluating these metrics have yet to be established. We propose a set of automatic and interpretable measures for assessing the characteristics of corpus-level semantic similarity metrics, allowing sensible comparison of their behavior. We demonstrate the effectiveness of our evaluation measures in capturing fundamental characteristics by evaluating them on a collection of classical and state-of-the-art metrics. Our measures revealed that recently-developed metrics are becoming better in identifying semantic distributional mismatch while classical metrics are more sensitive to perturbations in the surface text levels.