Can LLMs Play \^O \u{A}n Quan Game? A Study of Multi-Step Planning and Decision Making

TL;DR

This study evaluates large language models' ability to plan and strategize in the Vietnamese board game n Quan, revealing their strengths and limitations in multi-step decision making and strategic reasoning.

Contribution

It introduces a novel testbed using n Quan to assess LLMs' strategic planning and decision-making capabilities across different agent personas.

Findings

LLMs demonstrate varying strategic skills depending on model size.

Larger models perform better in multi-step planning and dynamic decision making.

Insights into LLMs' reasoning strengths and weaknesses in strategic contexts.

Abstract

In this paper, we explore the ability of large language models (LLMs) to plan and make decisions through the lens of the traditional Vietnamese board game, \^O \u{A}n Quan. This game, which involves a series of strategic token movements and captures, offers a unique environment for evaluating the decision-making and strategic capabilities of LLMs. Specifically, we develop various agent personas, ranging from aggressive to defensive, and employ the \^O \u{A}n Quan game as a testbed for assessing LLM performance across different strategies. Through experimentation with models like Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, and Llama-3.3-70B-Instruct, we aim to understand how these models execute strategic decision-making, plan moves, and manage dynamic game states. The results will offer insights into the strengths and weaknesses of LLMs in terms of reasoning and strategy, contributing to a deeper understanding of their general capabilities.