Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time

TL;DR

This paper introduces SITAlign, a novel inference-time framework for aligning large language models with human preferences by satisficing, which balances primary objectives with secondary thresholds, backed by theoretical bounds and empirical validation.

Contribution

It proposes a new satisficing-based inference method for LLM alignment, providing theoretical bounds and demonstrating superior performance over existing multi-objective strategies.

Findings

SITAlign outperforms state-of-the-art methods by 22.3% in helpfulness GPT-4 win-tie rate.

The framework effectively balances helpfulness and harmlessness constraints.

Theoretical bounds on sub-optimality are derived for the satisficing approach.

Abstract

Aligning large language models with humans is challenging due to the inherently multifaceted nature of preference feedback. While existing approaches typically frame this as a multi-objective optimization problem, they often overlook how humans actually make decisions. Research on bounded rationality suggests that human decision making follows satisficing strategies-optimizing primary objectives while ensuring others meet acceptable thresholds. To bridge this gap and operationalize the notion of satisficing alignment, we propose SITAlign: an inference time framework that addresses the multifaceted nature of alignment by maximizing a primary objective while satisfying threshold-based constraints on secondary criteria. We provide theoretical insights by deriving sub-optimality bounds of our satisficing based inference alignment approach. We empirically validate SITAlign's performance through extensive experimentation on multiple benchmarks. For instance, on the PKU-SafeRLHF dataset with the primary objective of maximizing helpfulness while ensuring a threshold on harmlessness, SITAlign outperforms the state-of-the-art multi objective decoding strategy by a margin of 22.3% in terms of GPT-4 win-tie rate for helpfulness reward while adhering to the threshold on harmlessness.