Viewpoint Matters: Dynamically Optimizing Viewpoints with Masked Autoencoder for Visual Manipulation

TL;DR

This paper introduces MAE-Select, a framework that dynamically chooses optimal viewpoints for robotic manipulation using masked autoencoder representations, enhancing single-camera systems and sometimes outperforming multi-camera setups.

Contribution

The paper presents a novel active viewpoint selection method leveraging pre-trained masked autoencoders, enabling adaptive, label-free viewpoint optimization in robotic systems.

Findings

MAE-Select improves robotic manipulation capabilities.

It can outperform multi-camera systems in certain scenarios.

The method does not require labeled viewpoints.

Abstract

Robotic manipulation continues to be a challenge, and imitation learning (IL) enables robots to learn tasks from expert demonstrations. Current IL methods typically rely on fixed camera setups, where cameras are manually positioned in static locations, imposing significant limitations on adaptability and coverage. Inspired by human active perception, where humans dynamically adjust their viewpoint to capture the most relevant and least noisy information, we propose MAE-Select, a novel framework for active viewpoint selection in single-camera robotic systems. MAE-Select fully leverages pre-trained multi-view masked autoencoder representations and dynamically selects the next most informative viewpoint at each time chunk without requiring labeled viewpoints. Extensive experiments demonstrate that MAE-Select improves the capabilities of single-camera systems and, in some cases, even surpasses multi-camera setups. The project will be available at https://mae-select.github.io.