VistaBot: View-Robust Robot Manipulation via Spatiotemporal-Aware View Synthesis

TL;DR

VistaBot is a novel framework that enhances robot manipulation robustness to camera viewpoint changes by integrating geometric models with diffusion-based view synthesis, enabling effective cross-view generalization without camera calibration.

Contribution

The paper introduces a geometry-aware view synthesis model and a latent action planner that improve cross-view generalization in robotic manipulation tasks.

Findings

VistaBot improves the View Generalization Score by over 2.7 times.

It achieves high-quality novel view synthesis in both simulation and real-world tasks.

The framework does not require camera calibration at test time.

Abstract

Recently, end-to-end robotic manipulation models have gained significant attention for their generalizability and scalability. However, they often suffer from limited robustness to camera viewpoint changes when training with a fixed camera. In this paper, we propose VistaBot, a novel framework that integrates feed-forward geometric models with video diffusion models to achieve view-robust closed-loop manipulation without requiring camera calibration at test time. Our approach consists of three key components: 4D geometry estimation, view synthesis latent extraction, and latent action learning. VistaBot is integrated into both action-chunking (ACT) and diffusion-based ($\pi_0$) policies and evaluated across simulation and real-world tasks. We further introduce the View Generalization Score (VGS) as a new metric for comprehensive evaluation of cross-view generalization. Results show that VistaBot improves VGS by 2.79$\times$ and 2.63$\times$ over ACT and $\pi_0$, respectively, while also achieving high-quality novel view synthesis. Our contributions include a geometry-aware synthesis model, a latent action planner, a new benchmark metric, and extensive validation across diverse environments. The code and models will be made publicly available.