PromptNCE: Pointwise Mutual Information Predictions Using Only LLMs and Contrastive Estimation Prompts

TL;DR

This paper introduces PromptNCE, a contrastive prompting method enabling large language models to estimate pointwise mutual information zero-shot, outperforming existing methods across multiple datasets.

Contribution

The paper proposes PromptNCE, a novel contrastive prompting approach that accurately estimates mutual information without task-specific training, applicable in low-data scenarios.

Findings

PromptNCE achieves Spearman correlation up to 0.82 with human PMI.

It outperforms five other information-theoretic estimators on three datasets.

Adding an OTHER category improves the accuracy of probability estimation.

Abstract

Estimating mutual information from text usually requires training a task-specific critic, which limits its use in low-data settings. We ask whether large language models can instead estimate pointwise mutual information zero-shot, using only prompts and elicited probabilities. We introduce a benchmark with human-derived ground-truth PMI across three publicly available datasets, and evaluate five information-theoretic prompting-based estimators. Our main method, PromptNCE, frames conditional probability estimation as a contrastive task and augments the candidate set with an explicit OTHER category. We show theoretically that adding OTHER recovers the true conditional P(y | x) rather than just a ranking over listed candidates, turning a contrastive prompt into a general-purpose zero-shot probability estimator. PromptNCE is the best zero-shot method on all three datasets, reaching Spearman correlation up to 0.82 with human-derived PMI. We also present a case study in computer science education showing how these estimators can be used to score student knowledge summaries in a low-data setting.