What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understanding

TL;DR

This paper introduces Task2Quiz, a new evaluation paradigm for assessing whether large language model agents truly understand their environment, revealing current limitations in memory and exploration strategies.

Contribution

The paper proposes Task2Quiz (T2Q), an environment understanding evaluation framework, and demonstrates its effectiveness through T2QBench, exposing gaps in current LLM agent capabilities.

Findings

Task success often does not indicate environment understanding.

Current memory mechanisms are ineffective for grounded environment modeling.

Proactive exploration and detailed state representation are key to improvement.

Abstract

Large language model (LLM) agents have demonstrated remarkable capabilities in complex decision-making and tool-use tasks, yet their ability to generalize across varying environments remains a under-examined concern. Current evaluation paradigms predominantly rely on trajectory-based metrics that measure task success, while failing to assess whether agents possess a grounded, transferable model of the environment. To address this gap, we propose Task-to-Quiz (T2Q), a deterministic and automated evaluation paradigm designed to decouple task execution from world-state understanding. We instantiate this paradigm in T2QBench, a suite comprising 30 environments and 1,967 grounded QA pairs across multiple difficulty levels. Our extensive experiments reveal that task success is often a poor proxy for environment understanding, and that current memory machanism can not effectively help agents acquire a grounded model of the environment. These findings identify proactive exploration and fine-grained state representation as primary bottlenecks, offering a robust foundation for developing more generalizable autonomous agents.