Case Repositories: Towards Case-Based Reasoning for AI Alignment

TL;DR

This paper introduces a case-based reasoning approach for AI alignment, constructing case repositories from expert input, AI-generated variations, and public judgment to guide AI behavior in complex societal contexts.

Contribution

It proposes a novel framework for AI alignment using case repositories, integrating human expertise, AI generation, and public input to handle societal value conflicts.

Findings

A process to assemble case repositories for AI alignment.

Use of LLMs to generate diverse case variations.

Engagement of the public to judge and refine cases.

Abstract

Case studies commonly form the pedagogical backbone in law, ethics, and many other domains that face complex and ambiguous societal questions informed by human values. Similar complexities and ambiguities arise when we consider how AI should be aligned in practice: when faced with vast quantities of diverse (and sometimes conflicting) values from different individuals and communities, with whose values is AI to align, and how should AI do so? We propose a complementary approach to constitutional AI alignment, grounded in ideas from case-based reasoning (CBR), that focuses on the construction of policies through judgments on a set of cases. We present a process to assemble such a case repository by: 1) gathering a set of ``seed'' cases -- questions one may ask an AI system -- in a particular domain, 2) eliciting domain-specific key dimensions for cases through workshops with domain experts, 3) using LLMs to generate variations of cases not seen in the wild, and 4) engaging with the public to judge and improve cases. We then discuss how such a case repository could assist in AI alignment, both through directly acting as precedents to ground acceptable behaviors, and as a medium for individuals and communities to engage in moral reasoning around AI.