Critic in the Loop: A Tri-System VLA Framework for Robust Long-Horizon Manipulation

TL;DR

This paper presents a hierarchical robotic manipulation framework combining fast reactive control, semantic reasoning, and adaptive scheduling to improve robustness and efficiency in long-horizon tasks.

Contribution

It introduces a novel Tri-System architecture with dynamic VLM-Expert scheduling and a visual Critic for adaptive control, reducing reliance on costly VLM inference.

Findings

Achieves state-of-the-art performance on long-horizon manipulation benchmarks.

Enhances robustness in out-of-distribution scenarios.

Reduces expensive VLM queries through intelligent scheduling.

Abstract

Balancing high-level semantic reasoning with low-level reactive control remains a core challenge in visual robotic manipulation. While Vision-Language Models (VLMs) excel at cognitive planning, their inference latency precludes real-time execution. Conversely, fast Vision-Language-Action (VLA) models often lack the semantic depth required for complex, long-horizon tasks. To bridge this gap, we introduce Critic in the Loop, an adaptive hierarchical framework driven by dynamic VLM-Expert scheduling. At its core is a bionic Tri-System architecture comprising a VLM brain for global reasoning, a VLA cerebellum for reactive execution, and a lightweight visual Critic. By continuously monitoring the workspace, the Critic dynamically routes control authority. It sustains rapid closed-loop execution via the VLA for routine subtasks, and adaptively triggers the VLM for replanning upon detecting execution anomalies such as task stagnation or failures. Furthermore, our architecture seamlessly integrates human-inspired rules to intuitively break infinite retry loops. This visually-grounded scheduling minimizes expensive VLM queries, while substantially enhancing system robustness and autonomy in out-of-distribution (OOD) scenarios. Comprehensive experiments on challenging, long-horizon manipulation benchmarks reveal that our approach achieves state-of-the-art performance.