Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding

TL;DR

This paper investigates how large-scale reasoning distillation enhances long-context understanding in language models, demonstrating significant improvements in multi-document question answering and addressing the 'lost in the middle' challenge.

Contribution

It provides a comprehensive analysis of the impact of reasoning distillation on long-context comprehension, highlighting its benefits for retrieval-augmented systems.

Findings

Distilled reasoning improves long-context information extraction.

Enhanced reasoning fosters better multi-document question answering.

Mitigates the 'lost in the middle' problem in long-context models.

Abstract

Reasoning distillation has emerged as an effective approach to enhance the reasoning capabilities of smaller language models. However, the impact of large-scale reasoning distillation on other critical abilities, particularly in-context retrieval and reasoning, remains unexplored. This gap in understanding is particularly significant given the increasing importance of Retrieval-Augmented Generation (RAG) systems, where efficient acquisition and utilization of contextual information are paramount for generating reliable responses. Motivated by the need to understand how the extended long-CoT process influences long-context comprehension, we conduct a comprehensive investigation using a series of open-source models distilled from Deepseek-R1, renowned for its exceptional reasoning capabilities. Our study focuses on evaluating these models' performance in extracting and integrating relevant information from extended contexts through multi-document question and answering tasks. Through rigorous experimentation, we demonstrate that distilled reasoning patterns significantly improve long-context understanding. Our analysis reveals that distillation fosters greater long-context awareness by promoting more detailed and explicit reasoning processes during context analysis and information parsing. This advancement effectively mitigates the persistent "lost in the middle" issue that has hindered long-context models.