Viewpoint-Agnostic Manipulation Policies with Strategic Vantage Selection

TL;DR

Vantage is a strategic viewpoint selection framework that fine-tunes pre-trained manipulation policies to be robust against viewpoint changes, using an information gain optimization approach with theoretical guarantees.

Contribution

It introduces a novel viewpoint selection method that enhances viewpoint-agnostic manipulation performance with minimal fine-tuning and theoretical convergence guarantees.

Findings

Increases success rate by 25% for diffusion policies.

Improves robustness in dynamic-camera settings.

Outperforms fixed, grid, or random viewpoint strategies.

Abstract

Since vision-based manipulation policies are typically trained from data gathered from a single viewpoint, their performance drops when the view changes during deployment. Naively aggregating demonstrations from numerous random views is not only costly but also known to destabilize learning, as excessive visual diversity acts as noise. We present Vantage, a viewpoint selection framework to fine-tune any pre-trained policy on a small, strategically set of camera poses to induce viewpoint-agnostic behavior. Instead of relying on costly brute-force search over viewpoints, Vantage formulates camera placement as an information gain optimization problem in a continuous space. This approach balances exploration of novel poses with exploitation of promising ones, while also providing theoretical guarantees about convergence and robustness. Across manipulation tasks and policy families, Vantage consistently improves success under viewpoint shifts compared to fixed, grid, or random data selection strategies with only a handful of fine-tuning steps. Experiments conducted on simulated and real-world setups show that Vantage increases the task success rate by 25% for diffusion policies, and yields robust gains in dynamic-camera settings.