MineNPC-Task: Task Suite for Memory-Aware Minecraft Agents

TL;DR

This paper introduces MineNPC-Task, a comprehensive benchmark for evaluating memory-aware Minecraft agents through real co-play tasks, emphasizing transparency, reproducibility, and in-world evidence-based assessment.

Contribution

It presents a novel, human-authored task suite with explicit dependencies and validators, enabling realistic and reproducible evaluation of memory and planning in Minecraft agents.

Findings

Identified common failure patterns in code, inventory, and navigation.

Demonstrated successful recovery through clarifications and memory use.

Participants rated interaction quality positively.

Abstract

We present MineNPC-Task, a user-authored benchmark and evaluation harness for testing memory-aware, mixed-initiative LLM agents in open-world Minecraft. Rather than relying on synthetic prompts, tasks are elicited through formative and summative co-play with expert players, then normalized into parametric templates with explicit preconditions and dependency structure. These tasks are paired with machine-checkable validators under a bounded-knowledge policy that forbids out-of-world shortcuts. The harness captures plan, action, and memory events, including plan previews, targeted clarifications, memory reads and writes, precondition checks, and repair attempts, and reports outcomes relative to the total number of attempted subtasks using only in-world evidence. As an initial snapshot, we instantiate the framework with GPT-4o and evaluate 216 subtasks across 8 experienced players. We observe recurring breakdown patterns in code execution, inventory and tool handling, referencing, and navigation, alongside successful recoveries supported by mixed-initiative clarifications and lightweight memory use. Participants rated interaction quality and interface usability positively, while noting the need for stronger memory persistence across tasks. We release the complete task suite, validators, logs, and evaluation harness to support transparent and reproducible evaluation of future memory-aware embodied agents.