Argumentatively Coherent Judgmental Forecasting

TL;DR

This paper introduces the concept of argumentative coherence in judgmental forecasting, demonstrating that enforcing coherence improves accuracy for humans and LLMs, but users often do not naturally adhere to this property.

Contribution

It formally defines argumentative coherence in forecasting and empirically shows its benefits for accuracy, highlighting the gap between theoretical coherence and user intuition.

Findings

Filtering incoherent forecasts improves accuracy for humans and LLMs.

Users generally do not naturally produce coherent forecasts.

Enforcing coherence can enhance the reliability of judgmental forecasting.

Abstract

Judgmental forecasting employs human opinions to make predictions about future events, rather than exclusively historical data as in quantitative forecasting. When these opinions form an argumentative structure around forecasts, it is useful to study the properties of the forecasts from an argumentative perspective. In this paper, we advocate and formally define a property of argumentative coherence, which, in essence, requires that a forecaster's reasoning is coherent with their forecast. We then conduct three evaluations with our notion of coherence. First, we assess the impact of enforcing coherence on human forecasters as well as on Large Language Model (LLM)-based forecasters, given that they have recently shown to be competitive with human forecasters. In both cases, we show that filtering out incoherent predictions improves forecasting accuracy consistently, supporting the practical value of coherence in both human and LLM-based forecasting. Then, via crowd-sourced user experiments, we show that, despite its apparent intuitiveness and usefulness, users do not generally align with this coherence property. This points to the need to integrate, within argumentation-based judgmental forecasting, mechanisms to filter out incoherent opinions before obtaining group forecasting predictions.