Harnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMs

TL;DR

This paper presents SALU, a novel LLM-based approach that integrates unanswerability detection within the model, improving reliability and reducing hallucinations in conversational information retrieval systems.

Contribution

SALU introduces a multi-task training and reinforcement learning framework that enables LLMs to recognize unanswerable questions and abstain appropriately, enhancing trustworthiness.

Findings

SALU outperforms baseline systems in accuracy for answering or abstaining.

Human evaluations show higher factuality and appropriate abstention.

SALU significantly reduces hallucinations in responses.

Abstract

Conversational Information Retrieval (CIR) systems, while offering intuitive access to information, face a significant challenge: reliably handling unanswerable questions to prevent the generation of misleading or hallucinated content. Traditional approaches often rely on external classifiers, which can introduce inconsistencies with the core generative Large Language Models (LLMs). This paper introduces Self-Aware LLM for Unanswerability (SALU), a novel approach that deeply integrates unanswerability detection directly within the LLM's generative process. SALU is trained using a multi-task learning framework for both standard Question Answering (QA) and explicit abstention generation for unanswerable queries. Crucially, it incorporates a confidence-score-guided reinforcement learning with human feedback (RLHF) phase, which explicitly penalizes hallucinated responses and rewards appropriate abstentions, fostering intrinsic self-awareness of knowledge boundaries. Through extensive experiments on our custom-built C-IR_Answerability dataset, SALU consistently outperforms strong baselines, including hybrid LLM-classifier systems, in overall accuracy for correctly answering or abstaining from questions. Human evaluation further confirms SALU's superior reliability, achieving high scores in factuality, appropriate abstention, and, most importantly, a dramatic reduction in hallucination, demonstrating its ability to robustly "know when to say 'I don't know'."