An Objective Laboratory Protocol for Evaluating Cognition of Non-Human Systems Against Human Cognition

TL;DR

This paper presents a new Bayesian laboratory protocol for objectively evaluating and comparing the cognitive abilities of non-human systems against humans, addressing limitations of previous methods.

Contribution

It introduces a comprehensive, Bayesian evaluation protocol that assesses generality and novelty in cognition, providing confidence measures and diagnostic insights.

Findings

Protocol successfully differentiates cognitive capabilities

Provides confidence levels for claims of human-like cognition

Identifies specific cognitive areas where systems fall short

Abstract

In this paper I describe and reduce to practice an objective protocol for evaluating the cognitive capabilities of a non-human system against human cognition in a laboratory environment. This is important because the existence of a non-human system with cognitive capabilities comparable to those of humans might make once-philosophical questions of safety and ethics immediate and urgent. Past attempts to devise evaluation methods, such as the Turing Test and many others, have not met this need; most of them either emphasize a single aspect of human cognition or a single theory of intelligence, fail to capture the human capacity for generality and novelty, or require success in the physical world. The protocol is broadly Bayesian, in that its primary output is a confidence statistic in relation to a claim. Further, it provides insight into the areas where and to what extent a particular system falls short of human cognition, which can help to drive further progress or precautions.