PMIScore: An Unsupervised Approach to Quantify Dialogue Engagement

TL;DR

PMIScore introduces an unsupervised method using mutual information to quantify dialogue engagement, providing a reliable, interpretable metric that can benchmark models and improve human-computer interactions.

Contribution

It presents a novel unsupervised approach to measure dialogue engagement using PMI, learned via divergence and neural networks, without requiring labeled data.

Findings

PMIScore accurately estimates PMI in synthetic and real datasets.

The PMI metric correlates well with perceived engagement levels.

PMIScore outperforms baseline methods in engagement quantification.

Abstract

High dialogue engagement is a crucial indicator of an effective conversation. A reliable measure of engagement could help benchmark large language models, enhance the effectiveness of human-computer interactions, or improve personal communication skills. However, quantifying engagement is challenging, since it is subjective and lacks a "gold standard". This paper proposes PMIScore, an efficient unsupervised approach to quantify dialogue engagement. It uses pointwise mutual information (PMI), which is the probability of generating a response conditioning on the conversation history. Thus, PMIScore offers a clear interpretation of engagement. As directly computing PMI is intractable due to the complexity of dialogues, PMIScore learned it through a dual form of divergence. The algorithm includes generating positive and negative dialogue pairs, extracting embeddings by large language models (LLMs), and training a small neural network using a mutual information loss function. We validated PMIScore on both synthetic and real-world datasets. Our results demonstrate the effectiveness of PMIScore in PMI estimation and the reasonableness of the PMI metric itself.