From Facts to Conclusions : Integrating Deductive Reasoning in Retrieval-Augmented LLMs

TL;DR

This paper introduces a unified reasoning framework for retrieval-augmented language models that enhances factual accuracy and interpretability by analyzing conflicts and providing justified answers, significantly improving performance.

Contribution

It proposes a novel reasoning-trace-augmented RAG framework with conflict analysis and a trust-score pipeline, addressing multiple issues simultaneously in retrieval-augmented LLMs.

Findings

End-to-end answer correctness improved from 0.069 to 0.883.

Behavioral adherence increased from 0.074 to 0.722.

Established a new dataset and evaluation pipeline for conflict-aware RAG.

Abstract

Retrieval-Augmented Generation (RAG) grounds large language models (LLMs) in external evidence, but fails when retrieved sources conflict or contain outdated or subjective information. Prior work address these issues independently but lack unified reasoning supervision. We propose a reasoning-trace-augmented RAG framework that adds structured, interpretable reasoning across three stages : (1) document-level adjudication, (2) conflict analysis, and (3) grounded synthesis, producing citation-linked answers or justified refusals. A Conflict-Aware Trust-Score (CATS) pipeline is introduced which evaluates groundedness, factual correctness, refusal accuracy, and conflict-behavior alignment using an LLM-as-a-Judge. Our 539-query reasoning dataset and evaluation pipeline establish a foundation for conflict-aware, interpretable RAG systems. Experimental results demonstrate substantial gains over baselines, most notably with Qwen, where Supervised Fine-Tuning improved End-to-End answer correctness from 0.069 to 0.883 and behavioral adherence from 0.074 to 0.722.