Opportunities and Risks of LLMs for Scalable Deliberation with Polis

TL;DR

This paper investigates how Large Language Models can enhance the Polis deliberation platform by improving moderation and summarization, while also addressing associated risks and future research directions.

Contribution

It demonstrates pilot experiments with LLMs like Claude to augment Polis, highlighting new summarization methods and discussing risks and mitigation strategies.

Findings

LLMs can effectively assist in moderation and summarization in Polis.

Summarization enables new collective meaning-making methods.

Context limitations of LLMs significantly affect result quality.

Abstract

Polis is a platform that leverages machine intelligence to scale up deliberative processes. In this paper, we explore the opportunities and risks associated with applying Large Language Models (LLMs) towards challenges with facilitating, moderating and summarizing the results of Polis engagements. In particular, we demonstrate with pilot experiments using Anthropic's Claude that LLMs can indeed augment human intelligence to help more efficiently run Polis conversations. In particular, we find that summarization capabilities enable categorically new methods with immense promise to empower the public in collective meaning-making exercises. And notably, LLM context limitations have a significant impact on insight and quality of these results. However, these opportunities come with risks. We discuss some of these risks, as well as principles and techniques for characterizing and mitigating them, and the implications for other deliberative or political systems that may employ LLMs. Finally, we conclude with several open future research directions for augmenting tools like Polis with LLMs.