clembench-2024: A Challenging, Dynamic, Complementary, Multilingual Benchmark and Underlying Flexible Framework for LLMs as Multi-Action Agents

TL;DR

This paper introduces clembench-2024, a flexible, multilingual benchmark framework for evaluating Large Language Models as multi-action agents through dynamic, interactive game-like environments, addressing current evaluation limitations.

Contribution

It presents a new adaptable framework for testing LLMs in interactive, multilingual settings, capable of evolving with new developments and avoiding data contamination.

Findings

Models perform below human levels in the benchmark.

The framework can adapt to new evaluation needs.

Prompting language significantly impacts model performance.

Abstract

It has been established in recent work that Large Language Models (LLMs) can be prompted to "self-play" conversational games that probe certain capabilities (general instruction following, strategic goal orientation, language understanding abilities), where the resulting interactive game play can be automatically scored. In this paper, we take one of the proposed frameworks for setting up such game-play environments, and further test its usefulness as an evaluation instrument, along a number of dimensions: We show that it can easily keep up with new developments while avoiding data contamination, we show that the tests implemented within it are not yet saturated (human performance is substantially higher than that of even the best models), and we show that it lends itself to investigating additional questions, such as the impact of the prompting language on performance. We believe that the approach forms a good basis for making decisions on model choice for building applied interactive systems, and perhaps ultimately setting up a closed-loop development environment of system and simulated evaluator.