Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

TL;DR

Retrieving and concatenating similar training instances to input significantly improves NLP task performance, achieving state-of-the-art results with a simple, cost-effective method that leverages training data more effectively.

Contribution

Introducing REINA, a simple retrieval-based approach that enhances NLP models by utilizing training data, leading to substantial performance gains across multiple tasks.

Findings

Significant performance improvements on summarization, translation, and QA tasks.

Achieved state-of-the-art results on XSum, BigPatent, and CommonsenseQA.

Simple retrieval method outperforms many existing approaches.

Abstract

Retrieval-based methods have been shown to be effective in NLP tasks via introducing external knowledge. However, the indexing and retrieving of large-scale corpora bring considerable computational cost. Surprisingly, we found that REtrieving from the traINing datA (REINA) only can lead to significant gains on multiple NLG and NLU tasks. We retrieve the labeled training instances most similar to the input text and then concatenate them with the input to feed into the model to generate the output. Experimental results show that this simple method can achieve significantly better performance on a variety of NLU and NLG tasks, including summarization, machine translation, language modeling, and question answering tasks. For instance, our proposed method achieved state-of-the-art results on XSum, BigPatent, and CommonsenseQA. Our code is released, https://github.com/microsoft/REINA .