Improvisational Games as a Benchmark for Social Intelligence of AI Agents: The Case of Connections

TL;DR

This paper introduces the Connections game as a benchmark to evaluate social intelligence and reasoning abilities of AI agents, emphasizing collaboration and understanding of other agents.

Contribution

The paper formalizes the Connections game as a new benchmark for assessing social intelligence in AI, highlighting skills like reasoning, knowledge retrieval, and social awareness.

Findings

Connections effectively tests AI's reasoning and social skills.

AI agents demonstrate social awareness through communication in the game.

The benchmark reveals limitations and strengths of current language models.

Abstract

We formally introduce a improvisational wordplay game called Connections to explore reasoning capabilities of AI agents. Playing Connections combines skills in knowledge retrieval, summarization and awareness of cognitive states of other agents. We show how the game serves as a good benchmark for social intelligence abilities of language model based agents that go beyond the agents' own memory and deductive reasoning and also involve gauging the understanding capabilities of other agents. Finally, we show how through communication with other agents in a constrained environment, AI agents must demonstrate social awareness and intelligence in games involving collaboration.