Rethinking with Retrieval: Faithful Large Language Model Inference

TL;DR

This paper introduces a lightweight, post-processing retrieval method called Rethinking with Retrieval (RR) that enhances large language models' reasoning and explanation faithfulness without additional training, using external knowledge based on chain-of-thought prompts.

Contribution

The paper proposes a novel retrieval-based post-processing approach for LLMs that improves reasoning accuracy and explanation faithfulness without requiring fine-tuning or extra training.

Findings

RR improves LLM performance on reasoning tasks

RR produces more faithful explanations

The method is effective across multiple reasoning domains

Abstract

Despite the success of large language models (LLMs) in various natural language processing (NLP) tasks, the stored knowledge in these models may inevitably be incomplete, out-of-date, or incorrect. This motivates the need to utilize external knowledge to assist LLMs. Unfortunately, current methods for incorporating external knowledge often require additional training or fine-tuning, which can be costly and may not be feasible for LLMs. To address this issue, we propose a novel post-processing approach, rethinking with retrieval (RR), which retrieves relevant external knowledge based on the decomposed reasoning steps obtained from the chain-of-thought (CoT) prompting. This lightweight approach does not require additional training or fine-tuning and is not limited by the input length of LLMs. We evaluate the effectiveness of RR through extensive experiments with GPT-3 on three complex reasoning tasks: commonsense reasoning, temporal reasoning, and tabular reasoning. Our results show that RR can produce more faithful explanations and improve the performance of LLMs.