The Forecast Critic: Leveraging Large Language Models for Poor Forecast Identification

TL;DR

This paper explores how large language models can be used to automatically monitor and critique forecast quality in retail, demonstrating promising results in identifying errors and incorporating contextual information without domain-specific training.

Contribution

It systematically evaluates LLMs for forecast monitoring, showing their ability to detect forecast errors and incorporate unstructured data, advancing automated forecast evaluation methods.

Findings

LLMs can reliably detect forecast errors with an F1 score of 0.88.

Multi-modal LLMs effectively incorporate contextual signals, achieving an F1 score of 0.84.

Techniques work on real-world retail data, identifying unreasonable forecasts with at least 10% higher sCRPS.

Abstract

Monitoring forecasting systems is critical for customer satisfaction, profitability, and operational efficiency in large-scale retail businesses. We propose The Forecast Critic, a system that leverages Large Language Models (LLMs) for automated forecast monitoring, taking advantage of their broad world knowledge and strong ``reasoning'' capabilities. As a prerequisite for this, we systematically evaluate the ability of LLMs to assess time series forecast quality, focusing on three key questions. (1) Can LLMs be deployed to perform forecast monitoring and identify obviously unreasonable forecasts? (2) Can LLMs effectively incorporate unstructured exogenous features to assess what a reasonable forecast looks like? (3) How does performance vary across model sizes and reasoning capabilities, measured across state-of-the-art LLMs? We present three experiments, including on both synthetic and real-world forecasting data. Our results show that LLMs can reliably detect and critique poor forecasts, such as those plagued by temporal misalignment, trend inconsistencies, and spike errors. The best-performing model we evaluated achieves an F1 score of 0.88, somewhat below human-level performance (F1 score: 0.97). We also demonstrate that multi-modal LLMs can effectively incorporate unstructured contextual signals to refine their assessment of the forecast. Models correctly identify missing or spurious promotional spikes when provided with historical context about past promotions (F1 score: 0.84). Lastly, we demonstrate that these techniques succeed in identifying inaccurate forecasts on the real-world M5 time series dataset, with unreasonable forecasts having an sCRPS at least 10% higher than that of reasonable forecasts. These findings suggest that LLMs, even without domain-specific fine-tuning, may provide a viable and scalable option for automated forecast monitoring and evaluation.