Language Models Benefit from Preparation with Elicited Knowledge

TL;DR

The paper introduces PREP, a simple prompting technique that enhances language models' question answering by eliciting relevant knowledge through a two-step process, improving accuracy across multiple datasets.

Contribution

The paper presents PREP, a novel prompting method that leverages two language model instances to better access relevant knowledge without domain-specific tuning.

Findings

PREP improves accuracy on multiple QA datasets.

PREP outperforms existing methods in zero-shot settings.

The approach is versatile across different question types.

Abstract

The zero-shot chain of thought (CoT) approach is often used in question answering (QA) by language models (LMs) for tasks that require multiple reasoning steps. However, some QA tasks hinge more on accessing relevant knowledge than on chaining reasoning steps. We introduce a simple prompting technique, called PREP, that involves using two instances of LMs: the first (LM1) generates relevant information, and the second (LM2) receives the information from the user and answers the question. This design is intended to make better use of the LM's instruction-following capability. PREP is applicable across various QA tasks without domain-specific prompt engineering. PREP is developed on a dataset of 100 QA questions, derived from an extensive schematic dataset specifying artifact parts and material composition. These questions ask which of two artifacts is less likely to share materials with another artifact. Such questions probe the LM's knowledge of shared materials in the part structure of different artifacts. We test our method on our parts-and-materials dataset and three published commonsense reasoning datasets. The average accuracy of our method is consistently higher than that of all the other tested methods across all the tested datasets.