High Accuracy, Less Talk (HALT): Reliable LLMs through Capability-Aligned Finetuning

TL;DR

HALT is a finetuning method for LLMs that improves reliability by enabling models to abstain from uncertain content, significantly increasing correctness while balancing response completeness across multiple domains.

Contribution

The paper introduces HALT, a novel post-training approach that aligns LLM capabilities with response confidence, reducing hallucinations and improving reliability.

Findings

Increases mean correctness of responses by 15% on average.

Improves F1 score by 4% compared to baselines.

Enhances correctness from 51% to 87% in a large Llama model.

Abstract

Large Language Models (LLMs) currently respond to every prompt. However, they can produce incorrect answers when they lack knowledge or capability -- a problem known as hallucination. We instead propose post-training an LLM to generate content only when confident in its correctness and to otherwise (partially) abstain. Specifically, our method, HALT, produces capability-aligned post-training data that encodes what the model can and cannot reliably generate. We generate this data by splitting responses of the pretrained LLM into factual fragments (atomic statements or reasoning steps), and use ground truth information to identify incorrect fragments. We achieve capability-aligned finetuning responses by either removing incorrect fragments or replacing them with "Unsure from Here" -- according to a tunable threshold that allows practitioners to trade off response completeness and mean correctness of the response's fragments. We finetune four open-source models for biography writing, mathematics, coding, and medicine with HALT for three different trade-off thresholds. HALT effectively trades off response completeness for correctness, increasing the mean correctness of response fragments by 15% on average, while resulting in a 4% improvement in the F1 score (mean of completeness and correctness of the response) compared to the relevant baselines. By tuning HALT for highest correctness, we train a single reliable Llama3-70B model with correctness increased from 51% to 87% across all four domains while maintaining 53% of the response completeness achieved with standard finetuning.