Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesis

TL;DR

This paper introduces Dream2Fix, a novel framework that synthesizes realistic failure scenarios from successful demonstrations to train robots for autonomous, precise recovery from manipulation errors, significantly improving correction accuracy.

Contribution

Dream2Fix creates a high-quality, paired failure-correction dataset using generative models and verification, enabling effective training of recovery policies without real-world data collection or simulation.

Findings

Achieves 81.3% correction accuracy, outperforming previous methods.

Enables zero-shot, closed-loop failure recovery in real robots.

Generates over 120,000 high-fidelity failure-correction samples.

Abstract

While recent foundation models have significantly advanced robotic manipulation, these systems still struggle to autonomously recover from execution errors. Current failure-learning paradigms rely on either costly and unsafe real-world data collection or simulator-based perturbations, which introduce a severe sim-to-real gap. Furthermore, existing visual analyzers predominantly output coarse, binary diagnoses rather than the executable, trajectory-level corrections required for actual recovery. To bridge the gap between failure diagnosis and actionable recovery, we introduce Dream2Fix, a framework that synthesizes photorealistic, counterfactual failure rollouts directly from successful real-world demonstrations. By perturbing actions within a generative world model, Dream2Fix creates paired failure-correction data without relying on simulators. To ensure the generated data is physically viable for robot learning, we implement a structured verification mechanism that strictly filters rollouts for task validity, visual coherence, and kinematic safety. This engine produces a high-fidelity dataset of over 120k paired samples. Using this dataset, we fine-tune a vision-language model to jointly predict failure types and precise recovery trajectories, mapping visual anomalies directly to corrective actions. Extensive real-world robotic experiments show our approach achieves state-of-the-art correction accuracy, improving from 19.7% to 81.3% over prior baselines, and successfully enables zero-shot closed-loop failure recovery in physical deployments.