Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning

TL;DR

This paper explores how instruction tuning on short contexts can generalize to longer contexts in large language models, introducing a data synthesis method to generate effective long-context training data, improving performance on document-level tasks.

Contribution

The paper introduces context synthesis, a novel framework using LLMs to generate long-context instruction data, enhancing long-context instruction tuning efficiency and effectiveness.

Findings

Models trained on short contexts can generalize to longer contexts.

Context synthesis outperforms previous instruction data generation methods.

Approach approaches human-annotated data performance on LongBench.

Abstract

Long-context modelling for large language models (LLMs) has been a key area of recent research because many real world use cases require reasoning over longer inputs such as documents. The focus of research into modelling long context has been on how to model position and there has been little investigation into other important aspects of language modelling such as instruction tuning. Long context training examples are challenging and expensive to create and use. In this paper, we investigate how to design instruction data for the post-training phase of a long context pre-trained model: how much and what type of context is needed for optimal and efficient post-training. Our controlled study reveals that models instruction-tuned on short contexts can effectively generalize to longer ones, while also identifying other critical factors such as instruction difficulty and context composition. Based on these findings, we propose context synthesis, a novel data synthesis framework that leverages off-the-shelf LLMs to generate extended background contexts for high-quality instruction-answer pairs. Experiment results on the document-level benchmark (LongBench) demonstrate that our proposed approach outperforms previous instruction synthesis approaches and comes close to the performance of human-annotated long-context instruction data. The project will be available at: https://github.com/NJUNLP/context-synthesis.