ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks

TL;DR

ALFRED is a comprehensive benchmark designed to evaluate and advance models that interpret natural language instructions into sequences of actions within realistic household environments, addressing the complexity of everyday tasks.

Contribution

The paper introduces ALFRED, a new benchmark with detailed, realistic tasks and a large dataset of demonstrations, to improve grounded language understanding in household scenarios.

Findings

Baseline models perform poorly on ALFRED, indicating room for improvement.

ALFRED's tasks are more complex than existing datasets, challenging current models.

The benchmark bridges the gap between research and real-world household task applications.

Abstract

We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks. ALFRED includes long, compositional tasks with non-reversible state changes to shrink the gap between research benchmarks and real-world applications. ALFRED consists of expert demonstrations in interactive visual environments for 25k natural language directives. These directives contain both high-level goals like "Rinse off a mug and place it in the coffee maker." and low-level language instructions like "Walk to the coffee maker on the right." ALFRED tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets. We show that a baseline model based on recent embodied vision-and-language tasks performs poorly on ALFRED, suggesting that there is significant room for developing innovative grounded visual language understanding models with this benchmark.