Trustworthy Human Computation: A Survey

TL;DR

This survey explores the foundations, challenges, and future directions for establishing trustworthiness in human computation systems, focusing on reliability, social trust, and human-AI collaboration.

Contribution

It provides a comprehensive framework for understanding trust in human computation, integrating technical and ethical perspectives, and outlines future research challenges.

Findings

Trustworthiness analyzed using the RAS analogy.

Discussion of social trustworthiness including fairness, privacy, transparency.

Highlights the importance of mutual trust in human-AI collaboration.

Abstract

Human computation is an approach to solving problems that prove difficult using AI only, and involves the cooperation of many humans. Because human computation requires close engagement with both "human populations as users" and "human populations as driving forces," establishing mutual trust between AI and humans is an important issue to further the development of human computation. This survey lays the groundwork for the realization of trustworthy human computation. First, the trustworthiness of human computation as computing systems, that is, trust offered by humans to AI, is examined using the RAS (Reliability, Availability, and Serviceability) analogy, which define measures of trustworthiness in conventional computer systems. Next, the social trustworthiness provided by human computation systems to users or participants is discussed from the perspective of AI ethics, including fairness, privacy, and transparency. Then, we consider human--AI collaboration based on two-way trust, in which humans and AI build mutual trust and accomplish difficult tasks through reciprocal collaboration. Finally, future challenges and research directions for realizing trustworthy human computation are discussed.