The Future of Open Human Feedback

TL;DR

This paper explores the potential for creating an open ecosystem for human feedback on language models, emphasizing community involvement, challenges, and sustainable practices to improve AI safety and capabilities.

Contribution

It provides an interdisciplinary assessment of open human feedback, identifies key challenges, reviews current approaches, and proposes a framework for a sustainable, community-driven feedback ecosystem.

Findings

Successful open practices from peer production and citizen science.

Main challenges include privacy, incentivization, and quality control.

Recommendations for building an open, sustainable feedback infrastructure.

Abstract

Human feedback on conversations with language language models (LLMs) is central to how these systems learn about the world, improve their capabilities, and are steered toward desirable and safe behaviors. However, this feedback is mostly collected by frontier AI labs and kept behind closed doors. In this work, we bring together interdisciplinary experts to assess the opportunities and challenges to realizing an open ecosystem of human feedback for AI. We first look for successful practices in peer production, open source, and citizen science communities. We then characterize the main challenges for open human feedback. For each, we survey current approaches and offer recommendations. We end by envisioning the components needed to underpin a sustainable and open human feedback ecosystem. In the center of this ecosystem are mutually beneficial feedback loops, between users and specialized models, incentivizing a diverse stakeholders community of model trainers and feedback providers to support a general open feedback pool.