Derivation Prompting: A Logic-Based Method for Improving Retrieval-Augmented Generation

TL;DR

Derivation Prompting is a logic-inspired technique that enhances retrieval-augmented generation by constructing interpretable derivation trees, reducing errors in knowledge-intensive question answering tasks.

Contribution

It introduces a novel, interpretable prompting method that systematically derives conclusions, improving accuracy in domain-specific question answering with retrieval-augmented models.

Findings

Significantly reduces unacceptable answers compared to traditional RAG.

Constructs interpretable derivation trees for better control over generation.

Improves accuracy in knowledge-intensive, domain-specific tasks.

Abstract

The application of Large Language Models to Question Answering has shown great promise, but important challenges such as hallucinations and erroneous reasoning arise when using these models, particularly in knowledge-intensive, domain-specific tasks. To address these issues, we introduce Derivation Prompting, a novel prompting technique for the generation step of the Retrieval-Augmented Generation framework. Inspired by logic derivations, this method involves deriving conclusions from initial hypotheses through the systematic application of predefined rules. It constructs a derivation tree that is interpretable and adds control over the generation process. We applied this method in a specific case study, significantly reducing unacceptable answers compared to traditional RAG and long-context window methods.