Flora: Effortless Context Construction to Arbitrary Length and Scale

TL;DR

Flora is a novel, human/LLM-free method for constructing long contexts by assembling short instructions, significantly improving long-context performance of LLMs while preserving short-context abilities.

Contribution

Flora introduces an effortless, scalable strategy for long-context construction that enhances LLM performance without requiring human or LLM interventions.

Findings

Outperforms existing methods on long-context benchmarks.

Maintains strong short-context performance.

Applicable to multiple LLM architectures.

Abstract

Effectively handling long contexts is challenging for Large Language Models (LLMs) due to the rarity of long texts, high computational demands, and substantial forgetting of short-context abilities. Recent approaches have attempted to construct long contexts for instruction tuning, but these methods often require LLMs or human interventions, which are both costly and limited in length and diversity. Also, the drop in short-context performances of present long-context LLMs remains significant. In this paper, we introduce Flora, an effortless (human/LLM-free) long-context construction strategy. Flora can markedly enhance the long-context performance of LLMs by arbitrarily assembling short instructions based on categories and instructing LLMs to generate responses based on long-context meta-instructions. This enables Flora to produce contexts of arbitrary length and scale with rich diversity, while only slightly compromising short-context performance. Experiments on Llama3-8B-Instruct and QwQ-32B show that LLMs enhanced by Flora excel in three long-context benchmarks while maintaining strong performances in short-context tasks. Our data-construction code is available at \href{https://github.com/txchen-USTC/Flora}{https://github.com/txchen-USTC/Flora}.