Improving the Distributional Alignment of LLMs using Supervision

TL;DR

This paper demonstrates that simple supervision techniques can enhance the alignment of large language models with diverse population groups across various subjective questions and datasets.

Contribution

It introduces a supervision method that improves distributional alignment of LLMs and provides a benchmark for future research across multiple datasets and models.

Findings

Supervision improves LLM alignment with diverse groups.

Alignment varies across specific population groups.

Benchmarking over many LLMs and prompts offers insights for future work.

Abstract

The ability to accurately align LLMs with diverse population groups on subjective questions would have great value. In this work, we show that adding simple supervision can more consistently improve the alignment of LLM-generated distributions with diverse population groups, as measured across three datasets spanning public health, public opinion, and values and beliefs. Beyond evaluating average alignment, we also report how alignment varies across specific groups. Our broad findings provide insights into the distributional alignment of LLM generations with diverse populations. By conducting evaluation over many LLMs and prompting strategies, we provide a benchmark to stimulate future research.