Know the Unknown: An Uncertainty-Sensitive Method for LLM Instruction Tuning

TL;DR

This paper introduces US-Tuning, a two-stage method that improves large language models' ability to recognize their knowledge limits and reduces hallucinations, leading to more accurate and faithful responses.

Contribution

The paper presents US-Tuning, a novel two-stage instruction tuning approach that enhances LLMs' awareness of their knowledge boundaries and adherence to instructions.

Findings

US-Tuning significantly reduces incorrect answers in contextual QA.

Fine-tuned Llama2-7B improves out-of-knowledge question handling by 34.7%.

Model outperforms GPT-4 by 4.2% in overall performance.

Abstract

Large language models (LLMs) demonstrate remarkable capabilities but face challenges from hallucinations, which typically arise from insufficient knowledge or context. While instructing LLMs to acknowledge knowledge limitations by responding with "I don't know" appears promising, we find that models consistently struggle with admitting knowledge gaps. This challenge may originate from current instruction datasets that emphasise answer generation over knowledge boundary awareness. To address this limitation, we introduce Uncertainty-and-Sensitivity-Aware Tuning (US-Tuning), a novel two-stage approach for contextual question answering (QA). The first stage enhances LLMs' ability to recognise their knowledge boundaries, while the second stage reinforces instruction adherence through carefully designed causal prompts. Our experimental results demonstrate that US-Tuning not only significantly reduces incorrect answers in contextual QA but also improves models' faithfulness to their parametric knowledge, mitigating hallucinations in general QA tasks. Our fine-tuned Llama2-7B model achieves up to a 34.7% improvement in handling out-of-knowledge questions and outperforms GPT-4 by 4.2% in overall performance.