Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback

TL;DR

This paper introduces a reinforcement learning framework that trains large language models to reject questions beyond their knowledge, thereby reducing hallucinations and improving response reliability.

Contribution

It proposes a novel RL-based method called RLKF that dynamically identifies knowledge boundaries and trains models to refuse out-of-knowledge questions.

Findings

RLKF significantly reduces hallucinations in LLMs.

Models trained with RLKF show improved accuracy on mathematical questions.

Rejection mechanisms enhance overall model reliability.

Abstract

Large Language Models (LLMs) often generate erroneous outputs, known as hallucinations, due to their limitations in discerning questions beyond their knowledge scope. While addressing hallucination has been a focal point in research, previous efforts primarily concentrate on enhancing correctness without giving due consideration to the significance of rejection mechanisms. In this paper, we conduct a comprehensive examination of the role of rejection, introducing the notion of model reliability along with corresponding metrics. These metrics measure the model's ability to provide accurate responses while adeptly rejecting questions exceeding its knowledge boundaries, thereby minimizing hallucinations. To improve the inherent reliability of LLMs, we present a novel alignment framework called Reinforcement Learning from Knowledge Feedback (RLKF). RLKF leverages knowledge feedback to dynamically determine the model's knowledge boundary and trains a reliable reward model to encourage the refusal of out-of-knowledge questions. Experimental results on mathematical questions affirm the substantial efficacy of RLKF in significantly enhancing LLM reliability.