Systematic Evaluation of Long-Context LLMs on Financial Concepts

TL;DR

This paper systematically evaluates the performance and limitations of long-context GPT-4 large language models on financial tasks, revealing brittleness at longer contexts and highlighting the need for more rigorous evaluation metrics.

Contribution

It provides a comprehensive assessment of LC LLMs on real-world financial tasks, identifying their fragility at longer contexts and proposing improved evaluation practices.

Findings

LC LLMs show performance decline at longer contexts

Models experience catastrophic failures with increased task complexity

Sensitivity to instruction placement and formatting affects outputs

Abstract

Long-context large language models (LC LLMs) promise to increase reliability of LLMs in real-world tasks requiring processing and understanding of long input documents. However, this ability of LC LLMs to reliably utilize their growing context windows remains under investigation. In this work, we evaluate the performance of state-of-the-art GPT-4 suite of LC LLMs in solving a series of progressively challenging tasks, as a function of factors such as context length, task difficulty, and position of key information by creating a real world financial news dataset. Our findings indicate that LC LLMs exhibit brittleness at longer context lengths even for simple tasks, with performance deteriorating sharply as task complexity increases. At longer context lengths, these state-of-the-art models experience catastrophic failures in instruction following resulting in degenerate outputs. Our prompt ablations also reveal unfortunate continued sensitivity to both the placement of the task instruction in the context window as well as minor markdown formatting. Finally, we advocate for more rigorous evaluation of LC LLMs by employing holistic metrics such as F1 (rather than recall) and reporting confidence intervals, thereby ensuring robust and conclusive findings.