Book your room in the Turing Hotel! A symmetric and distributed Turing Test with multiple AIs and humans

TL;DR

This paper introduces 'TuringHotel', a novel distributed Turing Test involving mixed communities of humans and large language models within a new platform, revealing that models still sometimes mimic humans convincingly and can be distinguished with difficulty.

Contribution

The paper presents a new community-based Turing Test framework and a platform for distributed AI-human interactions, expanding traditional Turing Test paradigms.

Findings

Models are sometimes confused with humans in the community setting.

Human fingerprints in language are still detectable but ambiguous.

Distributed experiments are feasible and valuable for monitoring AI evolution.

Abstract

In this paper, we report our experience with ``TuringHotel'', a novel extension of the Turing Test based on interactions within mixed communities of Large Language Models (LLMs) and human participants. The classical one-to-one interaction of the Turing Test is reinterpreted in a group setting, where both human and artificial agents engage in time-bounded discussions and, interestingly, are both judges and respondents. This community is instantiated in the novel platform UNaIVERSE (https://unaiverse.io), creating a ``World'' which defines the roles and interaction dynamics, facilitated by the platform's built-in programming tools. All communication occurs over an authenticated peer-to-peer network, ensuring that no third parties can access the exchange. The platform also provides a unified interface for humans, accessible via both mobile devices and laptops, that was a key component of the experience in this paper. Results of our experimentation involving 17 human participants and 19 LLMs revealed that current models are still sometimes confused as humans. Interestingly, there are several unexpected mistakes, suggesting that human fingerprints are still identifiable but not fully unambiguous, despite the high-quality language skills of artificial participants. We argue that this is the first experiment conducted in such a distributed setting, and that similar initiatives could be of national interest to support ongoing experiments and competitions aimed at monitoring the evolution of large language models over time.