Vision-Language-Policy Model for Dynamic Robot Task Planning

TL;DR

This paper introduces a vision-language-policy model that enables robots to interpret natural language instructions, reason over their environment, and adapt their task strategies dynamically, improving flexibility and generalization in real-world scenarios.

Contribution

It presents a novel vision-language model fine-tuned on real-world data that allows robots to interpret instructions, reason about scenes, and adapt their behavior policies dynamically during task execution.

Findings

Model effectively interprets semantic instructions.

Enables dynamic adjustment of task strategies.

Demonstrates strong generalization across different robots and tasks.

Abstract

Bridging the gap between natural language commands and autonomous execution in unstructured environments remains an open challenge for robotics. This requires robots to perceive and reason over the current task scene through multiple modalities, and to plan their behaviors to achieve their intended goals. Traditional robotic task-planning approaches often struggle to bridge low-level execution with high-level task reasoning, and cannot dynamically update task strategies when instructions change during execution, which ultimately limits their versatility and adaptability to new tasks. In this work, we propose a novel language model-based framework for dynamic robot task planning. Our Vision-Language-Policy (VLP) model, based on a vision-language model fine-tuned on real-world data, can interpret semantic instructions and integrate reasoning over the current task scene to generate behavior policies that control the robot to accomplish the task. Moreover, it can dynamically adjust the task strategy in response to changes in the task, enabling flexible adaptation to evolving task requirements. Experiments conducted with different robots and a variety of real-world tasks show that the trained model can efficiently adapt to novel scenarios and dynamically update its policy, demonstrating strong planning autonomy and cross-embodiment generalization. Videos: https://robovlp.github.io/