Beyond Na\"ive Prompting: Strategies for Improved Context-aided Forecasting with LLMs

TL;DR

This paper introduces a comprehensive framework with four strategies to enhance context-aided forecasting using large language models, addressing diagnostics, accuracy, and efficiency to improve practical deployment.

Contribution

It proposes a unified set of strategies for diagnosing, improving, and optimizing LLM-based forecasting, filling gaps in understanding and efficiency.

Findings

Diagnostic strategies reveal the 'Execution Gap' in model reasoning.

Accuracy improvements of 25-50% through targeted strategies.

Adaptive routing reduces inference costs while maintaining accuracy.

Abstract

Real-world forecasting requires models to integrate not only historical data but also relevant contextual information provided in textual form. While large language models (LLMs) show promise for context-aided forecasting, critical challenges remain: we lack diagnostic tools to understand failure modes, performance remains far below their potential, and high computational costs limit practical deployment. We introduce a unified framework of four strategies that address these limitations along three orthogonal dimensions: model diagnostics, accuracy, and efficiency. Through extensive evaluation across model families from small open-source models to frontier models including Gemini, GPT, and Claude, we uncover both fundamental insights and practical solutions. Our findings span three key dimensions: diagnostic strategies reveal the "Execution Gap" where models correctly explain how context affects forecasts but fail to apply this reasoning; accuracy-focused strategies achieve substantial performance improvements of 25-50%; and efficiency-oriented approaches show that adaptive routing between small and large models can approach large model accuracy on average while significantly reducing inference costs. These orthogonal strategies can be flexibly integrated based on deployment constraints, providing practitioners with a comprehensive toolkit for practical LLM-based context-aided forecasting.