The Imitation Game According To Turing

TL;DR

This paper critically examines claims that large language models can pass the Turing Test by conducting a rigorous, standards-adherent imitation game, and finds that current models do not meet the criteria, challenging hype about their intelligence.

Contribution

The study provides a detailed, standards-based Turing Test on GPT-4-Turbo, revealing it cannot convincingly imitate humans, thus questioning recent claims of passing the Turing Test.

Findings

GPT-4-Turbo was identified by most participants as non-human.

None of the tested models convincingly passed the rigorous Turing Test.

Claims that LLMs can think are unsupported by strict Turing Test evaluations.

Abstract

The current cycle of hype and anxiety concerning the benefits and risks to human society of Artificial Intelligence is fuelled, not only by the increasing use of generative AI and other AI tools by the general public, but also by claims made on behalf of such technology by popularizers and scientists. In particular, recent studies have claimed that Large Language Models (LLMs) can pass the Turing Test-a goal for AI since the 1950s-and therefore can "think". Large-scale impacts on society have been predicted as a result. Upon detailed examination, however, none of these studies has faithfully applied Turing's original instructions. Consequently, we conducted a rigorous Turing Test with GPT-4-Turbo that adhered closely to Turing's instructions for a three-player imitation game. We followed established scientific standards where Turing's instructions were ambiguous or missing. For example, we performed a Computer-Imitates-Human Game (CIHG) without constraining the time duration and conducted a Man-Imitates-Woman Game (MIWG) as a benchmark. All but one participant correctly identified the LLM, showing that one of today's most advanced LLMs is unable to pass a rigorous Turing Test. We conclude that recent extravagant claims for such models are unsupported, and do not warrant either optimism or concern about the social impact of thinking machines.