Not Yet: Humans Outperform LLMs in a Colonel Blotto Tournament

TL;DR

This study compares human and LLM strategies in the Colonel Blotto game, revealing humans outperform LLMs due to better-calibrated heuristics and strategic reasoning, with minimal strategy adjustment across tournaments.

Contribution

First comprehensive tournament-based comparison of human and LLM strategies in a complex game, highlighting the importance of strategic sophistication and reasoning depth.

Findings

Humans outperform LLMs in strategic allocation.

Strategic sophistication and reasoning depth are crucial for success.

Humans show minimal strategy adjustment across different opponents.

Abstract

The emergence of large language models (LLMs) has spurred economists to study how humans and LLMs behave in strategic settings. We organized a series of round-robin tournaments in the Colonel Blotto game. This game attracts game theorists' attention due to high-dimensional action space and the absence of pure strategy Nash equilibria. In the first tournament, more than 200 human participants competed against one another. In the second tournament, several popular LLMs were invited to submit strategies. In the third tournament, we matched the number of LLM strategies to the number submitted by humans. We find that humans more often employ better-calibrated intermediate-level allocation heuristics and outperform the simpler, more stereotyped strategies submitted by LLMs. Strategic sophistication is key to success if and only if the necessary level of reasoning depth is reached, while lower and higher levels of reasoning offer no clear advantage over the primitive strategies. Among humans, field of study weakly predicts success: participants with STEM backgrounds perform better in the first tournament. Surprisingly, humans almost do not adjust their strategies across tournaments with different sets of opponents. This result suggests that humans base their choices primarily on the game's rules rather than on the identity of their opponents, treating LLMs much like human competitors.