Ad-hoc Concept Forming in the Game Codenames as a Means for Evaluating Large Language Models

TL;DR

This paper uses the game Codenames as a novel benchmarking tool to assess large language models' linguistic and cognitive skills, revealing their strategies, limitations, and factors influencing performance.

Contribution

It introduces a new evaluation framework for LLMs based on gameplay, controlling variables like word type and opponent behavior to analyze model capabilities.

Findings

LLMs exhibit diverse strategies in Codenames gameplay.

Performance varies with word type and opponent speed.

The study identifies specific limitations of current LLMs.

Abstract

This study utilizes the game Codenames as a benchmarking tool to evaluate large language models (LLMs) with respect to specific linguistic and cognitive skills. LLMs play each side of the game, where one side generates a clue word covering several target words and the other guesses those target words. We designed various experiments by controlling the choice of words (abstract vs. concrete words, ambiguous vs. monosemic) or the opponent (programmed to be faster or slower in revealing words). Recent commercial and open-weight models were compared side-by-side to find out factors affecting their performance. The evaluation reveals details about their strategies, challenging cases, and limitations of LLMs.