CrafText Benchmark: Advancing Instruction Following in Complex Multimodal Open-Ended World

TL;DR

CrafText Benchmark is a comprehensive evaluation platform designed to assess an agent's ability to follow complex, diverse instructions in dynamic, multimodal environments, emphasizing generalization and adaptability.

Contribution

We introduce CrafText, a new benchmark with diverse instructions and dynamic tasks, along with an evaluation protocol for assessing generalization and adaptive decision-making.

Findings

Benchmark includes 3,924 instructions with 3,423 unique words.

Evaluates generalization to novel instructions and dynamic environments.

Provides a rigorous test of linguistic understanding and adaptability.

Abstract

Following instructions in real-world conditions requires the ability to adapt to the world's volatility and entanglement: the environment is dynamic and unpredictable, instructions can be linguistically complex with diverse vocabulary, and the number of possible goals an agent may encounter is vast. Despite extensive research in this area, most studies are conducted in static environments with simple instructions and a limited vocabulary, making it difficult to assess agent performance in more diverse and challenging settings. To address this gap, we introduce CrafText, a benchmark for evaluating instruction following in a multimodal environment with diverse instructions and dynamic interactions. CrafText includes 3,924 instructions with 3,423 unique words, covering Localization, Conditional, Building, and Achievement tasks. Additionally, we propose an evaluation protocol that measures an agent's ability to generalize to novel instruction formulations and dynamically evolving task configurations, providing a rigorous test of both linguistic understanding and adaptive decision-making.