Policy Prototyping for LLMs: Pluralistic Alignment via Interactive and Collaborative Policymaking

TL;DR

This paper introduces policy prototyping for large language models, enabling stakeholders to collaboratively draft and iteratively refine AI policies interactively, addressing limitations of linear, non-iterative alignment methods.

Contribution

It proposes a novel policy prototyping process inspired by design prototyping practices, facilitating interactive and collaborative policymaking for LLMs.

Findings

Demonstrated through a real-world LLM policymaking initiative

Outlined four guiding principles for policy prototyping

Highlights advantages over traditional linear alignment approaches

Abstract

Emerging efforts in AI alignment seek to broaden participation in shaping model behavior by eliciting and integrating collective input into a policy for model finetuning. While pluralistic, these processes are often linear and do not allow participating stakeholders to confirm whether potential outcomes of their contributions are indeed consistent with their intentions. Design prototyping has long advocated for rapid iteration using tight feedback loops of ideation, experimentation, and evaluation to mitigate these issues. We thus propose policy prototyping for LLMs, a new process that draws inspiration from prototyping practices to enable stakeholders to collaboratively and interactively draft LLM policies. Through learnings from a real-world LLM policymaking initiative at an industrial AI lab, we motivate our approach and characterize policy prototyping with four guiding principles. Because policy prototyping emphasizes a contrasting set of priorities compared to previous approaches, we envision our approach to be a valuable addition to the methodological repertoire for collaborative, pluralistic alignment.