Multimodal Behavior Tree Generation: A Small Vision-Language Model for Robot Task Planning

TL;DR

This paper introduces a multimodal vision-language model that generates behavior trees for robot task planning, leveraging a new dataset and fine-tuning techniques to achieve high success rates with reduced computational costs.

Contribution

It presents a novel dataset linking visual observations and instructions to behavior trees and fine-tunes a compact VLM to perform robotic task planning effectively.

Findings

Achieves 87% success rate in household tasks

Approaches performance of state-of-the-art models

Uses significantly fewer computational resources

Abstract

Large and small language models have been widely used for robotic task planning. At the same time, vision-language models (VLMs) have successfully tackled problems such as image captioning, scene understanding, and visual question answering. In this work, we combine these two approaches by deploying a compact, open-source multimodal model to generate behavior trees for robotic task planning. The main obstacle to achieving this goal is the lack of an existing dataset that links visual observations and instructions to executable behavior trees. We propose a method to construct such a dataset starting from existing robotic episodes (i.e., Open X-Embodiment), in which a large model serves as a teacher in a multi-stage generation pipeline. We use this dataset to fine-tune VLMs ranging from 500M to 4B parameters via parameter-efficient fine-tuning (PEFT). The generated behavior trees, compatible with the BehaviorTree.CPP library, are evaluated both offline, using structural and lexical metrics, and online through the execution of household tasks in a state-of-the-art embodied simulator. Our results demonstrate that our fine-tuned 4B-parameter VLM approaches the performance of state-of-the-art closed-source models, achieving an 87\% success rate while requiring only a fraction of the computational resources.