Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulation

TL;DR

This paper introduces an asynchronous active vision-action model for robotic manipulation that actively adjusts camera viewpoints and manipulation poses using few-shot reinforcement learning, improving performance under visual constraints.

Contribution

The paper presents a novel task-driven asynchronous model combining camera and manipulation policies trained with few-shot RL for better handling occlusions and limited views.

Findings

Outperforms baseline algorithms on 8 manipulation tasks

Effectively adjusts camera viewpoints to improve task success

Demonstrates robustness under visual constraints

Abstract

In real-world scenarios, many robotic manipulation tasks are hindered by occlusions and limited fields of view, posing significant challenges for passive observation-based models that rely on fixed or wrist-mounted cameras. In this paper, we investigate the problem of robotic manipulation under limited visual observation and propose a task-driven asynchronous active vision-action model.Our model serially connects a camera Next-Best-View (NBV) policy with a gripper Next-Best Pose (NBP) policy, and trains them in a sensor-motor coordination framework using few-shot reinforcement learning. This approach allows the agent to adjust a third-person camera to actively observe the environment based on the task goal, and subsequently infer the appropriate manipulation actions.We trained and evaluated our model on 8 viewpoint-constrained tasks in RLBench. The results demonstrate that our model consistently outperforms baseline algorithms, showcasing its effectiveness in handling visual constraints in manipulation tasks.