CodeGraphVLP: Code-as-Planner Meets Semantic-Graph State for Non-Markovian Vision-Language-Action Models

TL;DR

CodeGraphVLP introduces a hierarchical approach combining semantic graphs, code-based planning, and visual prompts to improve long-horizon robot manipulation in complex, partially observable environments.

Contribution

It presents a novel framework that maintains task-relevant information and guides visual reasoning, significantly enhancing performance over existing VLA models in non-Markovian tasks.

Findings

Improves task completion rates on real-world non-Markovian tasks.

Reduces planning latency compared to VLM-in-the-loop methods.

Demonstrates the effectiveness of semantic graphs and progress-guided prompting.

Abstract

Vision-Language-Action (VLA) models promise generalist robot manipulation, but are typically trained and deployed as short-horizon policies that assume the latest observation is sufficient for action reasoning. This assumption breaks in non-Markovian long-horizon tasks, where task-relevant evidence can be occluded or appear only earlier in the trajectory, and where clutter and distractors make fine-grained visual grounding brittle. We present CodeGraphVLP, a hierarchical framework that enables reliable long-horizon manipulation by combining a persistent semantic-graph state with an executable code-based planner and progress-guided visual-language prompting. The semantic-graph maintains task-relevant entities and relations under partial observability. The synthesized planner executes over this semantic-graph to perform efficient progress checks and outputs a subtask instruction together with subtask-relevant objects. We use these outputs to construct clutter-suppressed observations that focus the VLA executor on critical evidence. On real-world non-Markovian tasks, CodeGraphVLP improves task completion over strong VLA baselines and history-enabled variants while substantially lowering planning latency compared to VLM-in-the-loop planning. We also conduct extensive ablation studies to confirm the contributions of each component.