An Uncommon Task: Participatory Design in Legal AI

TL;DR

This paper examines a historic participatory AI design process in the legal domain, highlighting how collaborative approaches between computer scientists and lawyers influenced legal automation and informed current AI development critiques.

Contribution

It provides an empirical analysis of a decade-old participatory design process in legal AI, illustrating its structure, impact, and relevance to modern AI development debates.

Findings

Participatory approaches facilitated collaboration between lawyers and computer scientists.

Interactive simulation methods helped bridge research and real-world legal practice.

The case informs current critiques and calls for increased participation in AI development.

Abstract

Despite growing calls for participation in AI design, there are to date few empirical studies of what these processes look like and how they can be structured for meaningful engagement with domain experts. In this paper, we examine a notable yet understudied AI design process in the legal domain that took place over a decade ago, the impact of which still informs legal automation efforts today. Specifically, we examine the design and evaluation activities that took place from 2006 to 2011 within the TeXT Retrieval Conference's (TREC) Legal Track, a computational research venue hosted by the National Institute of Standards and Technologies. The Legal Track of TREC is notable in the history of AI research and practice because it relied on a range of participatory approaches to facilitate the design and evaluation of new computational techniques--in this case, for automating attorney document review for civil litigation matters. Drawing on archival research and interviews with coordinators of the Legal Track of TREC, our analysis reveals how an interactive simulation methodology allowed computer scientists and lawyers to become co-designers and helped bridge the chasm between computational research and real-world, high-stakes litigation practice. In analyzing this case from the recent past, our aim is to empirically ground contemporary critiques of AI development and evaluation and the calls for greater participation as a means to address them.