EVLP:Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuning

TL;DR

EVLP introduces a unified multimodal framework that integrates linguistic reasoning and visual generation for long-horizon embodied manipulation tasks, improving task planning through dynamic pretraining and reinforced fine-tuning.

Contribution

The paper presents a novel unified generation framework with dynamic perception pretraining and reinforced fine-tuning for multimodal embodied planning, addressing inconsistencies in prior methods.

Findings

Effective multimodal planning for long-horizon tasks achieved

Enhanced multimodal correlation through dynamic alignment

Improved spatio-visual reasoning in embodied tasks

Abstract

In complex embodied long-horizon manipulation tasks, effective task decomposition and execution require synergistic integration of textual logical reasoning and visual-spatial imagination to ensure efficient and accurate operation. Current methods fail to adopt a unified generation framework for multimodal planning, lead to inconsistent in multimodal planning. To address this challenge, we present \textbf{EVLP (Embodied Vision-Language Planner)}, an innovative multimodal unified generation framework that jointly models linguistic reasoning and visual generation. Our approach achieves multimodal planning for long-horizon tasks through a novel training pipeline incorporating dynamic pretraining and reinforced alignment. Our core innovations consist of three key components: \textbf{1) Unified Multimodal Generation Framework}: For understanding, We integrate semantic information with spatial features to provide comprehensive visual perception. For generation, we directly learn the joint distribution of discrete images for one-step visual synthesis, enabling coordinated language-visual modeling through learnable cross-modal attention mechanisms. \textbf{2) Dynamic Perception Pretraining}: We propose a bidirectional dynamic alignment strategy employing inverse dynamics tasks and forward dynamics tasks, effectively strengthening multimodal correlations within a unified feature space. \textbf{3) Reinforced Supervised Fine-Tuning}: While conducting instruction-based fine-tuning in the unified generation space, we construct a reinforce loss to align the spatial logic between textual actions and generated images, enabling the model to acquire spatio-awared multimodal planning capabilities.