HELM: Harness-Enhanced Long-horizon Memory for Vision-Language-Action Manipulation

TL;DR

HELM is a framework that significantly improves long-horizon vision-language manipulation by addressing memory, verification, and recovery gaps with specialized modules, outperforming existing models on multiple benchmarks.

Contribution

The paper introduces HELM, a model-agnostic framework with novel components like episodic memory, a learned verifier, and a rollback controller, to enhance long-horizon manipulation success.

Findings

HELM improves success rate by 23.1 percentage points on LIBERO-LONG.

The learned verifier outperforms rule-based and ensemble baselines.

HELM enhances recovery success under perturbations.

Abstract

Vision-Language-Action (VLA) models fail systematically on long-horizon manipulation tasks despite strong short-horizon performance. We show that this failure is not resolved by extending context length alone in the current reactive execution setting; instead, it stems from three recurring execution-loop deficiencies: the memory gap, the verification gap, and the recovery gap. We present HELM, a model-agnostic framework that addresses these deficiencies with three components: an Episodic Memory Module (EMM) that retrieves key task history via CLIP-indexed keyframes, a learned State Verifier (SV) that predicts action failure before execution from observation, action, subgoal, and memory-conditioned context, and a Harness Controller (HC) that performs rollback and replanning. The SV is the core learning contribution: it consistently outperforms rule-based feasibility checks and ensemble uncertainty baselines, and its effectiveness depends critically on access to episodic memory. On LIBERO-LONG, HELM improves task success rate by 23.1 percentage points over OpenVLA (58.4% to 81.5%), while extending the context window to H=32 yields only a 5.4-point gain and same-budget LoRA adaptation remains 12.2 points below HELM. HELM also improves long-horizon performance on CALVIN and substantially boosts recovery success under controlled perturbations. Ablations and mechanism analyses isolate the contribution of each component, and we release LIBERO-Recovery as a perturbation-injection protocol for evaluating failure recovery in long-horizon manipulation.