MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Models

TL;DR

MARVL introduces a multi-stage guidance framework that fine-tunes vision-language models for improved robotic manipulation, enhancing reward alignment, spatial grounding, and task understanding in reinforcement learning.

Contribution

It proposes a novel multi-stage guidance method that fine-tunes VLMs for better task decomposition and spatial-semantic consistency, advancing robotic RL reward design.

Findings

MARVL outperforms existing VLM-based reward methods on Meta-World benchmark.

It demonstrates higher sample efficiency in sparse-reward manipulation tasks.

MARVL shows increased robustness in complex robotic manipulation scenarios.

Abstract

Designing dense reward functions is pivotal for efficient robotic Reinforcement Learning (RL). However, most dense rewards rely on manual engineering, which fundamentally limits the scalability and automation of reinforcement learning. While Vision-Language Models (VLMs) offer a promising path to reward design, naive VLM rewards often misalign with task progress, struggle with spatial grounding, and show limited understanding of task semantics. To address these issues, we propose MARVL-Multi-stAge guidance for Robotic manipulation via Vision-Language models. MARVL fine-tunes a VLM for spatial and semantic consistency and decomposes tasks into multi-stage subtasks with task direction projection for trajectory sensitivity. Empirically, MARVL significantly outperforms existing VLM-reward methods on the Meta-World benchmark, demonstrating superior sample efficiency and robustness on sparse-reward manipulation tasks.