Beyond Viewpoint Generalization: What Multi-View Demonstrations Offer and How to Synthesize Them for Robot Manipulation?

TL;DR

This paper systematically studies the benefits of multi-view demonstrations for robot manipulation, revealing performance improvements, underlying mechanisms, and proposing RoboNVS to synthesize multi-view data from monocular inputs.

Contribution

It quantifies multi-view data benefits, analyzes underlying mechanisms, and introduces RoboNVS for synthesizing multi-view videos to enhance robot manipulation.

Findings

Multi-view demonstrations improve success and generalization in robot manipulation.

Performance benefits vary with view coverage, not always increasing with more views.

RoboNVS effectively synthesizes multi-view data from monocular inputs, boosting downstream performance.

Abstract

Does multi-view demonstration truly improve robot manipulation, or merely enhance cross-view robustness? We present a systematic study quantifying the performance gains, scaling behavior, and underlying mechanisms of multi-view data for robot manipulation. Controlled experiments show that, under both fixed and randomized backgrounds, multi-view demonstrations consistently improve single-view policy success and generalization. Performance varies non-monotonically with view coverage, revealing effective regimes rather than a simple "more is better" trend. Notably, multi-view data breaks the scaling limitation of single-view datasets and continues to raise performance ceilings after saturation. Mechanistic analysis shows that multi-view learning promotes manipulation-relevant visual representations, better aligns the action head with the learned feature distribution, and reduces overfitting. Motivated by the importance of multi-view data and its scarcity in large-scale robotic datasets, as well as the difficulty of collecting additional viewpoints in real world settings, we propose RoboNVS, a geometry-aware self-supervised framework that synthesizes novel-view videos from monocular inputs. The generated data consistently improves downstream policies in both simulation and real-world environments.