SPRI: Aligning Large Language Models with Context-Situated Principles

TL;DR

SPRI is a framework that automatically generates real-time, context-specific guiding principles for large language models, improving alignment, performance, and truthfulness with minimal human effort.

Contribution

It introduces SPRI, a novel method for automatic, instance-specific principle generation to enhance LLM alignment and performance.

Findings

SPRI achieves on-par performance with expert-crafted principles in complex tasks.

SPRI-generated principles outperform prior LLM-as-a-judge frameworks.

Using SPRI for synthetic data improves truthfulness significantly.

Abstract

Aligning Large Language Models to integrate and reflect human values, especially for tasks that demand intricate human oversight, is arduous since it is resource-intensive and time-consuming to depend on human expertise for context-specific guidance. Prior work has utilized predefined sets of rules or principles to steer the behavior of models (Bai et al., 2022; Sun et al., 2023). However, these principles tend to be generic, making it challenging to adapt them to each individual input query or context. In this work, we present Situated-PRInciples (SPRI), a framework requiring minimal or no human effort that is designed to automatically generate guiding principles in real-time for each input query and utilize them to align each response. We evaluate SPRI on three tasks, and show that 1) SPRI can derive principles in a complex domain-specific task that leads to on-par performance as expert-crafted ones; 2) SPRI-generated principles lead to instance-specific rubrics that outperform prior LLM-as-a-judge frameworks; 3) using SPRI to generate synthetic SFT data leads to substantial improvement on truthfulness. We release our code and model generations at https://github.com/honglizhan/SPRI-public.