Measuring Forecasting Skill from Text

TL;DR

This paper investigates how linguistic features in predictive texts relate to forecasting skill, demonstrating that language analysis can predict forecasters' accuracy across geopolitical and financial domains.

Contribution

It introduces linguistic metrics to analyze prediction texts and shows that forecasting skill can be accurately predicted solely from language features.

Findings

Linguistic factors correlate with forecasting accuracy

Language-based models can predict forecasting skill effectively

Skilled forecasters tend to use specific linguistic patterns

Abstract

People vary in their ability to make accurate predictions about the future. Prior studies have shown that some individuals can predict the outcome of future events with consistently better accuracy. This leads to a natural question: what makes some forecasters better than others? In this paper we explore connections between the language people use to describe their predictions and their forecasting skill. Datasets from two different forecasting domains are explored: (1) geopolitical forecasts from Good Judgment Open, an online prediction forum and (2) a corpus of company earnings forecasts made by financial analysts. We present a number of linguistic metrics which are computed over text associated with people's predictions about the future including: uncertainty, readability, and emotion. By studying linguistic factors associated with predictions, we are able to shed some light on the approach taken by skilled forecasters. Furthermore, we demonstrate that it is possible to accurately predict forecasting skill using a model that is based solely on language. This could potentially be useful for identifying accurate predictions or potentially skilled forecasters earlier.