Dynamic Forecasting of Conversation Derailment

TL;DR

This paper explores forecasting conversation derailment in online discussions using pretrained language models and dynamic training paradigms, highlighting benefits and challenges depending on data quality.

Contribution

It introduces a dynamic training approach for forecasting conversation derailment and evaluates its effectiveness with pretrained language encoders.

Findings

Pretrained language encoders outperform earlier methods.

Dynamic training can extend forecast horizon in high-quality data.

Dynamic training may harm performance in noisy, low-quality data.

Abstract

Online conversations can sometimes take a turn for the worse, either due to systematic cultural differences, accidental misunderstandings, or mere malice. Automatically forecasting derailment in public online conversations provides an opportunity to take early action to moderate it. Previous work in this space is limited, and we extend it in several ways. We apply a pretrained language encoder to the task, which outperforms earlier approaches. We further experiment with shifting the training paradigm for the task from a static to a dynamic one to increase the forecast horizon. This approach shows mixed results: in a high-quality data setting, a longer average forecast horizon can be achieved at the cost of a small drop in F1; in a low-quality data setting, however, dynamic training propagates the noise and is highly detrimental to performance.