Real2Sim via Active Perception with Behavior Trees Automatically Generated by VLMs

TL;DR

This paper introduces an autonomous framework that uses Vision-Language Models to generate behavior trees for efficient and safe physical parameter estimation in simulation environments based on natural language requests.

Contribution

It presents a novel intent-driven Real2Sim approach that leverages VLMs and Behavior Trees for automatic, minimal, and safe physical parameter acquisition in simulation.

Findings

Accurately estimates object mass, surface geometry, and friction.

Achieves significant efficiency improvements over exhaustive methods.

BT hierarchy acts as a safety filter against hallucinations.

Abstract

Constructing physically accurate simulation environments (Real2Sim) traditionally relies on manual system identification or rigid, exhaustive exploration routines. These task-agnostic pipelines often fail to leverage semantic scene context, leading to redundant physical interactions and inefficient data acquisition. In this paper, we present an autonomous, intent-driven Real2Sim framework that leverages Vision-Language Models (VLMs) for Semantic Task Decomposition. Given a high-level natural language request, an incomplete simulation description, and a visual observation, the framework autonomously identifies the minimal subset of missing physical parameters required for the simulation task. It then generates a reactive Behavior Tree (BT) composed of atomic motion and sensing primitives to selectively acquire these parameters through contact-rich robotic interaction. Extensive real-world experiments on a torque-controlled Franka Emika Panda demonstrate that our approach accurately estimates object mass, surface geometry, and derived parameters such as friction. Quantitative evaluations reveal significant operational efficiency gains compared to exhaustive baseline methods, while ablation studies confirm the robustness of the prompt architecture across different state-of-the-art VLMs. Furthermore, the reactive hierarchy of the BT acts as a deterministic safety filter, successfully mitigating generative VLM hallucinations and preventing unsafe physical anomalies. Ultimately, this work provides a scalable, efficient, and interpretable pipeline for building physics-aware digital twins directly from unstructured human intent.