Eliciting Critical Reasoning in Retrieval-Augmented Language Models via Contrastive Explanations

TL;DR

This paper introduces C-RAG, a framework that enhances retrieval-augmented language models by eliciting critical reasoning through contrastive explanations, leading to improved accuracy and robustness with fewer prompts.

Contribution

The paper presents a novel contrastive reasoning framework for RAG models that improves performance and robustness while reducing prompt requirements.

Findings

C-RAG outperforms state-of-the-art RAG models in experiments.

C-RAG requires fewer prompts and demonstrations.

C-RAG is robust to perturbations in retrieved documents.

Abstract

Retrieval-augmented generation (RAG) has emerged as a critical mechanism in contemporary NLP to support Large Language Models(LLMs) in systematically accessing richer factual context. However, the integration of RAG mechanisms brings its inherent challenges, as LLMs need to deal with potentially noisy contexts. Recent studies have shown that LLMs still struggle to critically analyse RAG-based in-context information, a limitation that may lead to incorrect inferences and hallucinations. In this paper, we investigate how to elicit critical reasoning in RAG via contrastive explanations. In particular, we propose Contrastive-RAG (C-RAG), a framework that (i) retrieves relevant documents given a query, (ii) selects and exemplifies relevant passages, and (iii) generates explanations that explicitly contrast the relevance of the passages to (iv) support the final answer. We show the impact of C-RAG building contrastive reasoning demonstrations from LLMs to instruct smaller models for retrieval-augmented tasks. Extensive experiments demonstrate that C-RAG improves state-of-the-art RAG models while (a) requiring significantly fewer prompts and demonstrations and (b) being robust to perturbations in the retrieved documents.