Decoding Alignment: A Critical Survey of LLM Development Initiatives through Value-setting and Data-centric Lens

TL;DR

This paper critically surveys recent LLM development initiatives, focusing on how they set values and utilize data for alignment, revealing the socio-technical challenges and practices across leading organizations.

Contribution

It provides a comprehensive audit of publicly available documentation from major LLM projects, emphasizing the value-setting and data-centric aspects of alignment practices.

Findings

Diverse approaches to value-setting across initiatives

Limited transparency in proprietary models' alignment processes

Highlighting socio-technical challenges in alignment practices

Abstract

AI Alignment, primarily in the form of Reinforcement Learning from Human Feedback (RLHF), has been a cornerstone of the post-training phase in developing Large Language Models (LLMs). It has also been a popular research topic across various disciplines beyond Computer Science, including Philosophy and Law, among others, highlighting the socio-technical challenges involved. Nonetheless, except for the computational techniques related to alignment, there has been limited focus on the broader picture: the scope of these processes, which primarily rely on the selected objectives (values), and the data collected and used to imprint such objectives into the models. This work aims to reveal how alignment is understood and applied in practice from a value-setting and data-centric perspective. For this purpose, we investigate and survey (`audit') publicly available documentation released by 6 LLM development initiatives by 5 leading organizations shaping this technology, focusing on proprietary (OpenAI's GPT, Anthropic's Claude, Google's Gemini) and open-weight (Meta's Llama, Google's Gemma, and Alibaba's Qwen) initiatives, all published in the last 3 years. The findings are documented in detail per initiative, while there is also an overall summary concerning different aspects, mainly from a value-setting and data-centric perspective. On the basis of our findings, we discuss a series of broader related concerns.