How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?

TL;DR

This paper investigates how large language models balance honesty and helpfulness, revealing that reinforcement learning enhances both, while prompting techniques influence their trade-offs, with GPT-4 Turbo showing human-like response patterns.

Contribution

The study applies psychological models to analyze LLMs' handling of honesty-helpfulness trade-offs and demonstrates how different training and prompting methods influence these behaviors.

Findings

Reinforcement learning from human feedback improves honesty and helpfulness.

Chain-of-thought prompting skews models towards helpfulness over honesty.

GPT-4 Turbo exhibits human-like sensitivity to context and framing.

Abstract

In day-to-day communication, people often approximate the truth - for example, rounding the time or omitting details - in order to be maximally helpful to the listener. How do large language models (LLMs) handle such nuanced trade-offs? To address this question, we use psychological models and experiments designed to characterize human behavior to analyze LLMs. We test a range of LLMs and explore how optimization for human preferences or inference-time reasoning affects these trade-offs. We find that reinforcement learning from human feedback improves both honesty and helpfulness, while chain-of-thought prompting skews LLMs towards helpfulness over honesty. Finally, GPT-4 Turbo demonstrates human-like response patterns including sensitivity to the conversational framing and listener's decision context. Our findings reveal the conversational values internalized by LLMs and suggest that even these abstract values can, to a degree, be steered by zero-shot prompting.