MDCure: A Scalable Pipeline for Multi-Document Instruction-Following

TL;DR

MDCure is a scalable framework that generates high-quality multi-document instruction data to improve large language models' performance on multi-document tasks without extensive pre-training or human annotation.

Contribution

We introduce MDCure, a novel data generation pipeline and reward model that enhance LLMs' multi-document capabilities efficiently and effectively.

Findings

MDCure improves LLM performance on multi-document benchmarks by up to 75.1%.

The framework enables small open-source models to outperform larger proprietary models.

MDCure's synthetic data generation is cost-effective and compatible with various models and training methods.

Abstract

Multi-document (MD) processing is crucial for LLMs to handle real-world tasks such as summarization and question-answering across large sets of documents. While LLMs have improved at processing long inputs, MD contexts still present unique difficulties, including management of inter-document dependencies, redundancy, and incoherent structures. To address this challenge, we introduce MDCure, a scalable and effective instruction data generation framework to enhance the MD capabilities of LLMs without the computational cost of pre-training or reliance on human-annotated data. MDCure generates high-quality synthetic MD instruction data over sets of articles via targeted prompts. We also introduce MDCureRM, a cost-effective, MD-specific reward model to score and filter generated data based on their training utility for MD settings. MDCure is compatible with open- and closed-source models in addition to policy optimization methods such as PPO, enabling even small open-source models to surpass proprietary LLMs as strong generators of high-quality MD instruction data without further data filtering. With MDCure, we fine-tune a wide variety of LLMs up to 70B parameters in size from the FlanT5, Qwen2, and LLAMA3.1 model families. Extensive evaluations on a wide range of MD and long-context benchmarks spanning various tasks and domains show MDCure consistently improves performance over pre-trained baselines and base models by up to 75.1%. Our code, datasets, and models are available at https://github.com/yale-nlp/MDCure.