Cross-Domain Demo-to-Code via Neurosymbolic Counterfactual Reasoning

TL;DR

This paper presents NeSyCR, a neurosymbolic framework that improves cross-domain robotic programming by reasoning about procedural differences, leading to significant success rate improvements in manipulation tasks.

Contribution

NeSyCR introduces a novel neurosymbolic counterfactual reasoning approach for verifiable cross-domain adaptation in video-instructed robotic programming.

Findings

Achieves 31.14% higher task success than baseline.

Effectively adapts to both simulated and real-world tasks.

Provides verifiable procedural revisions for domain mismatches.

Abstract

Recent advances in Vision-Language Models (VLMs) have enabled video-instructed robotic programming, allowing agents to interpret video demonstrations and generate executable control code. We formulate video-instructed robotic programming as a cross-domain adaptation problem, where perceptual and physical differences between demonstration and deployment induce procedural mismatches. However, current VLMs lack the procedural understanding needed to reformulate causal dependencies and achieve task-compatible behavior under such domain shifts. We introduce NeSyCR, a neurosymbolic counterfactual reasoning framework that enables verifiable adaptation of task procedures, providing a reliable synthesis of code policies. NeSyCR abstracts video demonstrations into symbolic trajectories that capture the underlying task procedure. Given deployment observations, it derives counterfactual states that reveal cross-domain incompatibilities. By exploring the symbolic state space with verifiable checks, NeSyCR proposes procedural revisions that restore compatibility with the demonstrated procedure. NeSyCR achieves a 31.14% improvement in task success over the strongest baseline Statler, showing robust cross-domain adaptation across both simulated and real-world manipulation tasks.