AgenticLab: A Real-World Robot Agent Platform that Can See, Think, and Act

TL;DR

AgenticLab is a comprehensive robot agent platform that integrates perception, reasoning, and action, enabling real-world manipulation in unstructured environments and providing a benchmark to evaluate VLM-based robot agents.

Contribution

It introduces a model-agnostic, closed-loop robot platform and benchmark for open-world manipulation, highlighting failure modes of current VLM-based agents in real-world tasks.

Findings

Identified failure modes in multi-step grounding and occlusion handling.

Revealed limitations of offline vision-language tests for real-robot tasks.

Provided a reproducible hardware and software stack for evaluation.

Abstract

Recent advances in large vision-language models (VLMs) have demonstrated generalizable open-vocabulary perception and reasoning, yet their real-robot manipulation capability remains unclear for long-horizon, closed-loop execution in unstructured, in-the-wild environments. Prior VLM-based manipulation pipelines are difficult to compare across different research groups' setups, and many evaluations rely on simulation, privileged state, or specially designed setups. We present AgenticLab, a model-agnostic robot agent platform and benchmark for open-world manipulation. AgenticLab provides a closed-loop agent pipeline for perception, task decomposition, online verification, and replanning. Using AgenticLab, we benchmark state-of-the-art VLM-based agents on real-robot tasks in unstructured environments. Our benchmark reveals several failure modes that offline vision-language tests (e.g., VQA and static image understanding) fail to capture, including breakdowns in multi-step grounding consistency, object grounding under occlusion and scene changes, and insufficient spatial reasoning for reliable manipulation. We will release the full hardware and software stack to support reproducible evaluation and accelerate research on general-purpose robot agents.