Alignment for Honesty

TL;DR

This paper emphasizes the importance of aligning large language models with honesty, proposing metrics, benchmarks, and training methods to improve truthful responses without overly conservative behavior.

Contribution

It introduces a formal definition of honesty for LLMs, develops metrics and benchmarks to measure it, and presents a flexible fine-tuning framework to enhance honesty.

Findings

Aligned models show increased honesty according to proposed metrics

The training framework maintains performance on other tasks

Open-source resources facilitate future research

Abstract

Recent research has made significant strides in aligning large language models (LLMs) with helpfulness and harmlessness. In this paper, we argue for the importance of alignment for \emph{honesty}, ensuring that LLMs proactively refuse to answer questions when they lack knowledge, while still not being overly conservative. However, a pivotal aspect of alignment for honesty involves discerning an LLM's knowledge boundaries, which demands comprehensive solutions in terms of metric development, benchmark creation, and training methodologies. We address these challenges by first establishing a precise problem definition and defining ``honesty'' inspired by the Analects of Confucius. This serves as a cornerstone for developing metrics that effectively measure an LLM's honesty by quantifying its progress post-alignment. Furthermore, we introduce a flexible training framework which is further instantiated by several efficient fine-tuning techniques that emphasize honesty without sacrificing performance on other tasks. Our extensive experiments reveal that these aligned models show a marked increase in honesty, as indicated by our proposed metrics. We open-source all relevant resources to facilitate future research at \url{https://github.com/GAIR-NLP/alignment-for-honesty}.