Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning

TL;DR

RetroPrompt enhances prompt learning by integrating a retrieval mechanism that decouples knowledge from memorization, leading to improved generalization and performance in few-shot and zero-shot NLP tasks.

Contribution

The paper introduces RetroPrompt, a novel retrieval-augmented prompt learning method that decouples knowledge from memorization to improve generalization in NLP.

Findings

RetroPrompt outperforms vanilla prompt learning in few-shot and zero-shot settings.

It reduces reliance on memorization, enhancing model generalization.

RetroPrompt shows consistent improvements across multiple datasets.

Abstract

Prompt learning approaches have made waves in natural language processing by inducing better few-shot performance while they still follow a parametric-based learning paradigm; the oblivion and rote memorization problems in learning may encounter unstable generalization issues. Specifically, vanilla prompt learning may struggle to utilize atypical instances by rote during fully-supervised training or overfit shallow patterns with low-shot data. To alleviate such limitations, we develop RetroPrompt with the motivation of decoupling knowledge from memorization to help the model strike a balance between generalization and memorization. In contrast with vanilla prompt learning, RetroPrompt constructs an open-book knowledge-store from training instances and implements a retrieval mechanism during the process of input, training and inference, thus equipping the model with the ability to retrieve related contexts from the training corpus as cues for enhancement. Extensive experiments demonstrate that RetroPrompt can obtain better performance in both few-shot and zero-shot settings. Besides, we further illustrate that our proposed RetroPrompt can yield better generalization abilities with new datasets. Detailed analysis of memorization indeed reveals RetroPrompt can reduce the reliance of language models on memorization; thus, improving generalization for downstream tasks. Code is available in https://github.com/zjunlp/PromptKG/tree/main/research/RetroPrompt.