Multicultural Spyfall: Assessing LLMs through Dynamic Multilingual Social Deduction Game

TL;DR

This paper introduces a dynamic, multilingual social deduction game framework called Multicultural Spyfall to evaluate LLMs' cultural and strategic reasoning, revealing significant performance gaps in non-English contexts and offering a scalable, leakage-resistant benchmarking alternative.

Contribution

It presents a novel game-based benchmarking method for multilingual and multicultural LLM evaluation, addressing limitations of static benchmarks and highlighting cultural challenges in non-English language understanding.

Findings

Models perform worse in non-English contexts with culturally specific entities.

Game-based rankings align with existing benchmarks like Chatbot Arena.

The approach offers a scalable, leakage-resistant evaluation method.

Abstract

The rapid advancement of Large Language Models (LLMs) has necessitated more robust evaluation methods that go beyond static benchmarks, which are increasingly prone to data saturation and leakage. In this paper, we propose a dynamic benchmarking framework for evaluating multilingual and multicultural capabilities through the social deduction game Spyfall. In our setup, models must engage in strategic dialogue to either identify a secret agent or avoid detection, utilizing culturally relevant locations or local foods. Our results show that our game-based rankings align closely with the Chatbot Arena. However, we find a significant performance gap in non-English contexts: models are generally less proficient when handling locally specific entities and often struggle with rule-following or strategic integrity in non-English languages. We demonstrate that this game-based approach provides a scalable, leakage-resistant, and culturally nuanced alternative to traditional NLP benchmarks. The game history can be accessed here https://huggingface.co/datasets/haryoaw/cultural-spyfall.