Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning

TL;DR

This paper introduces a supervised Chain-of-Thought approach with a new synthetic dataset, LongFinanceQA, to enhance long-context understanding in LLMs, demonstrating significant performance improvements in financial reasoning tasks.

Contribution

It presents a novel supervised CoT reasoning method, a synthetic dataset LongFinanceQA, and the Property-based Agentic Inference framework to improve long-context reasoning in LLMs.

Findings

GPT-4o-mini with PAI outperforms standard GPT-4o-mini by 20%.

Fine-tuned LLaMA-3.1-8B-Instruct gains 28% on Loong's financial subset.

Supervised CoT and synthetic data improve long-context understanding.

Abstract

Recent advances in Large Language Models (LLMs) have enabled them to process increasingly longer sequences, ranging from 2K to 2M tokens and even beyond. However, simply extending the input sequence length does not necessarily lead to effective long-context understanding. In this study, we integrate Chain-of-Thought (CoT) reasoning into LLMs in a supervised manner to facilitate effective long-context understanding. To achieve this, we introduce LongFinanceQA, a synthetic dataset in the financial domain designed to improve long-context reasoning. Unlike existing long-context synthetic data, LongFinanceQA includes intermediate CoT reasoning before the final conclusion, which encourages LLMs to perform explicit reasoning, improving accuracy and interpretability in long-context understanding. To generate synthetic CoT reasoning, we propose Property-based Agentic Inference (PAI), an agentic framework that simulates human-like reasoning steps, including property extraction, retrieval, and summarization. We evaluate PAI's reasoning capabilities by assessing GPT-4o-mini w/ PAI on the Loong benchmark, outperforming standard GPT-4o-mini by 20.0%. Furthermore, we fine-tune LLaMA-3.1-8B-Instruct on LongFinanceQA, achieving a 28.0% gain on Loong's financial subset.