Efficiently Aligning Language Models with Online Natural Language Feedback

TL;DR

This paper introduces methods for aligning language models in complex domains using online natural language feedback, significantly reducing the need for extensive expert supervision.

Contribution

It develops techniques to optimize language models with minimal expert supervision by iteratively updating proxy reward models using in-context learning and fine-tuning.

Findings

ICL methods recover up to 35% of performance with 50x fewer samples.

Fine-tuning recovers 80-100% of performance with 3-30x fewer samples.

Online natural language feedback improves data efficiency of supervision.

Abstract

Reinforcement learning with verifiable rewards has been used to elicit impressive performance from language models in many domains. But, broadly beneficial deployments of AI may require us to train models with strong capabilities in "fuzzy", hard-to-supervise domains. In this paper, we develop methods to align language models in fuzzy domains where human experts are still able to provide high-quality supervision signal, but only for a small number of model outputs, using online natural language feedback. Specifically, we train models by iteratively optimizing against proxy reward signals, stopping at the point of over-optimization, collecting fresh expert supervision, and updating the proxy reward. We construct proxy reward models from language models using in-context learning (ICL) and fine-tuning. We test our methods by eliciting creative writing and alignment research capabilities in Qwen3-8B and Haiku 4.5 respectively. For Qwen3-8B, ICL methods recover up to 35% of performance with 50x fewer expert samples, while fine-tuning methods recover 80% with up to 20x fewer samples and 100% with 3x fewer samples. For Haiku 4.5, ICL methods recover up to 35% of performance with 30x fewer samples, and fine-tuning methods recover 100% with 10x fewer samples. Our results suggest that online natural language feedback can substantially improve the data efficiency of expert supervision.