Balancing Truthfulness and Informativeness with Uncertainty-Aware Instruction Fine-Tuning

TL;DR

This paper explores the challenge of balancing truthfulness and informativeness in instruction fine-tuning of large language models, proposing two new methods to improve truthfulness and manage uncertainty.

Contribution

It introduces two novel IFT paradigms, $UNIT_{cut}$ and $UNIT_{ref}$, to enhance truthfulness and uncertainty recognition in LLMs, addressing the trade-off issue.

Findings

$UNIT_{cut}$ improves model truthfulness.

$UNIT_{ref}$ maintains informativeness and reduces hallucinations.

Unfamiliar knowledge in datasets can harm truthfulness.

Abstract

Instruction fine-tuning (IFT) can increase the informativeness of large language models (LLMs), but may reduce their truthfulness. This trade-off arises because IFT steers LLMs to generate responses containing long-tail knowledge that was not well covered during pre-training. As a result, models become more informative but less accurate when generalizing to unseen tasks. In this paper, we empirically demonstrate how unfamiliar knowledge in IFT datasets can negatively affect the truthfulness of LLMs, and we introduce two new IFT paradigms, $UNIT_{cut}$ and $UNIT_{ref}$, to address this issue. $UNIT_{cut}$ identifies and removes unfamiliar knowledge from IFT datasets to mitigate its impact on model truthfulness, whereas $UNIT_{ref}$ trains LLMs to recognize their uncertainty and explicitly indicate it at the end of their responses. Our experiments show that $UNIT_{cut}$ substantially improves LLM truthfulness, while $UNIT_{ref}$ maintains high informativeness and reduces hallucinations by distinguishing between confident and uncertain statements.