Wisdom of the Crowd, Without the Crowd: A Socratic LLM for Asynchronous Deliberation on Perspectivist Data

TL;DR

This paper introduces a Socratic dialogue system using LLMs to facilitate asynchronous deliberation in data annotation, improving accuracy and preserving diverse perspectives without the need for synchronous crowd discussions.

Contribution

It presents a novel LLM-based Socratic system that replaces synchronous crowd deliberation, enabling scalable, asynchronous preservation of diverse data perspectives.

Findings

Higher annotation accuracy in the Relation detection task.

Participants engaged in reasoned arguments and found the system well-received.

The approach encourages consideration of alternative perspectives.

Abstract

Data annotation underpins the success of modern AI, but the aggregation of crowd-collected datasets can harm the preservation of diverse perspectives in data. Difficult and ambiguous tasks cannot easily be collapsed into unitary labels. Prior work has shown that deliberation and discussion improve data quality and preserve diverse perspectives -- however, synchronous deliberation through crowdsourcing platforms is time-intensive and costly. In this work, we create a Socratic dialog system using Large Language Models (LLMs) to act as a deliberation partner in place of other crowdworkers. Against a benchmark of synchronous deliberation on two tasks (Sarcasm and Relation detection), our Socratic LLM encouraged participants to consider alternate annotation perspectives, update their labels as needed (with higher confidence), and resulted in higher annotation accuracy (for the Relation task where ground truth is available). Qualitative findings show that our agent's Socratic approach was effective at encouraging reasoned arguments from our participants, and that the intervention was well-received. Our methodology lays the groundwork for building scalable systems that preserve individual perspectives in generating more representative datasets.