{\lambda}: A Benchmark for Data-Efficiency in Long-Horizon Indoor Mobile Manipulation Robotics

TL;DR

This paper introduces the LAMBDA benchmark to evaluate data efficiency in long-horizon indoor mobile manipulation tasks, emphasizing realistic, human-collected data and comparing learning and neuro-symbolic methods.

Contribution

The paper presents a new benchmark with a realistic dataset for evaluating data efficiency in long-horizon manipulation, and compares learning and neuro-symbolic approaches.

Findings

Neuro-symbolic methods outperform pure learning methods in success rates.

Pretrained learning models still struggle with data efficiency.

The benchmark enables realistic evaluation of manipulation strategies.

Abstract

Learning to execute long-horizon mobile manipulation tasks is crucial for advancing robotics in household and workplace settings. However, current approaches are typically data-inefficient, underscoring the need for improved models that require realistically sized benchmarks to evaluate their efficiency. To address this, we introduce the LAMBDA ({\lambda}) benchmark-Long-horizon Actions for Mobile-manipulation Benchmarking of Directed Activities-which evaluates the data efficiency of models on language-conditioned, long-horizon, multi-room, multi-floor, pick-and-place tasks using a dataset of manageable size, more feasible for collection. Our benchmark includes 571 human-collected demonstrations that provide realism and diversity in simulated and real-world settings. Unlike planner-generated data, these trajectories offer natural variability and replay-verifiability, ensuring robust learning and evaluation. We leverage {\lambda} to benchmark current end-to-end learning methods and a modular neuro-symbolic approach that combines foundation models with task and motion planning. We find that learning methods, even when pretrained, yield lower success rates, while a neuro-symbolic method performs significantly better and requires less data.