Conversations Gone Awry, But Then? Evaluating Conversational Forecasting Models

TL;DR

This paper introduces a standardized evaluation framework and benchmark for conversational forecasting models, specifically for predicting conversation derailment, incorporating a new metric for forecast revision capabilities.

Contribution

It presents the first uniform evaluation framework and benchmark for CGA models, enabling reliable comparison and assessment of recent language modeling advancements.

Findings

Established a comprehensive benchmark for CGA models

Introduced a novel metric for forecast revision accuracy

Provided an overview of progress in conversational forecasting

Abstract

We often rely on our intuition to anticipate the direction of a conversation. Endowing automated systems with similar foresight can enable them to assist human-human interactions. Recent work on developing models with this predictive capacity has focused on the Conversations Gone Awry (CGA) task: forecasting whether an ongoing conversation will derail. In this work, we revisit this task and introduce the first uniform evaluation framework, creating a benchmark that enables direct and reliable comparisons between different architectures. This allows us to present an up-to-date overview of the current progress in CGA models, in light of recent advancements in language modeling. Our framework also introduces a novel metric that captures a model's ability to revise its forecast as the conversation progresses.