ImpScore: A Learnable Metric For Quantifying The Implicitness Level of Sentence

TL;DR

ImpScore is a novel, reference-free scalar metric that quantifies the implicitness level of language, enhancing analysis of model comprehension and revealing limitations in understanding implicit content.

Contribution

This paper introduces ImpScore, a learnable, interpretable regression-based metric for measuring language implicitness without external references, trained on contrastive pairs.

Findings

ImpScore correlates strongly with human judgments on implicitness.

It reveals current models' limitations in understanding highly implicit content.

Validated through user studies and applied to hate speech detection datasets.

Abstract

Handling implicit language is essential for natural language processing systems to achieve precise text understanding and facilitate natural interactions with users. Despite its importance, the absence of a metric for accurately measuring the implicitness of language significantly constrains the depth of analysis possible in evaluating models' comprehension capabilities. This paper addresses this gap by developing a scalar metric that quantifies the implicitness level of language without relying on external references. Drawing on principles from traditional linguistics, we define "implicitness" as the divergence between semantic meaning and pragmatic interpretation. To operationalize this definition, we introduce ImpScore, a reference-free metric formulated through an interpretable regression model. This model is trained using pairwise contrastive learning on a specially curated dataset consisting of (implicit sentence, explicit sentence) pairs. We validate ImpScore through a user study that compares its assessments with human evaluations on out-of-distribution data, demonstrating its accuracy and strong correlation with human judgments. Additionally, we apply ImpScore to hate speech detection datasets, illustrating its utility and highlighting significant limitations in current large language models' ability to understand highly implicit content. Our metric is publicly available at https://github.com/audreycs/ImpScore.