Learning Structured Robot Policies from Vision-Language Models via Synthetic Neuro-Symbolic Supervision

TL;DR

This paper presents a neuro-symbolic method that enables large vision-language models to generate interpretable, structured robot policies from visual and language inputs, trained via synthetic data, and successfully transferred to real robots.

Contribution

It introduces a scalable pipeline for generating synthetic multimodal datasets and demonstrates how a 12B-parameter model can learn structured policies for robotic manipulation.

Findings

Structured policies achieve zero-shot transfer to real robots.

Synthetic data generation bypasses the data bottleneck in robotic planning.

Neuro-symbolic policies outperform end-to-end visuomotor approaches in interpretability.

Abstract

Vision-Language Models (VLMs) have recently demonstrated strong capabilities in mapping multimodal observations to robot behaviors. However, most current approaches rely on end-to-end visuomotor policies that remain opaque and difficult to analyze, limiting their use in real-world robotic applications. In contrast, classical robotic systems often rely on structured policy representations that provide interpretability, modularity, and reactive execution. This work investigates how foundation models can be specialized to generate structured robot policies grounded in multimodal perception, bridging high-dimensional learning and symbolic control. We propose a neuro-symbolic approach in which a VLM synthesizes executable Behavior Tree policies from visual observations, natural language instructions, and structured system specifications. To enable scalable supervision without manual annotation, we introduce an automated pipeline that generates a synthetic multimodal dataset of domain-randomized scenes paired with instruction-policy examples produced by a foundation model. By decoupling structured task decomposition under constrained symbolic grammars from hardware-specific motor control, we demonstrate that a 12B-parameter model can learn structured spatial-symbolic mappings required for executable BT synthesis, solely through in-silico supervision. Real-world physical experiments on two heterogeneous robotic manipulators confirm that these structurally constrained policies achieve zero-shot transfer to real-world environments. The results emphasize that the data bottleneck in robotic planning can be bypassed by procedurally synthesizing high-fidelity, neuro-symbolic training data.