InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining

TL;DR

This paper introduces Retro 48B, the largest retrieval-augmented language model pretrained with retrieval, which significantly improves perplexity and zero-shot task performance after instruction tuning, demonstrating the scalability and effectiveness of retrieval augmentation.

Contribution

The paper presents Retro 48B, the largest retrieval-augmented pretrained language model, and shows its superior performance and efficiency, along with insights into architecture simplification.

Findings

Retro 48B outperforms GPT 43B in perplexity with minimal additional GPU hours.

InstructRetro improves zero-shot task performance by 7-16% over GPT.

Using only the decoder backbone yields comparable results, simplifying architecture.

Abstract

Pretraining auto-regressive large language models~(LLMs) with retrieval demonstrates better perplexity and factual accuracy by leveraging external databases. However, the size of existing pretrained retrieval-augmented LLM is still limited (e.g., Retro has 7.5B parameters), which limits the effectiveness of instruction tuning and zero-shot generalization. In this work, we introduce Retro 48B, the largest LLM pretrained with retrieval. Specifically, we continue to pretrain a 43B GPT model on additional 100 billion tokens using the Retro augmentation method by retrieving from 1.2 trillion tokens. Notably, the obtained foundation model, Retro 48B, largely outperforms the counterpart GPT 43B trained on 1.2T tokens in terms of perplexity with only 2.58% additional GPU hours, demonstrating the significant scaling potential of the method. After instruction tuning on Retro, InstructRetro demonstrates significant improvement over the instruction tuned GPT on a wide range of zero-shot tasks. Specifically, the average improvement of InstructRetro is 7% over its GPT counterpart across 8 short-form QA and reading comprehension tasks, 10% over GPT across 4 challenging long-form QA tasks, and 16% over GPT across 3 summarization tasks. Surprisingly, we find that one can ablate the encoder from InstructRetro architecture and directly use its decoder backbone, while achieving comparable results. Our results highlight the promising direction to obtain a better GPT decoder through continued pretraining with retrieval before instruction tuning. Our code and checkpoints are publicly available at: https://huggingface.co/nvidia/retro-48b-instruct-4k.