BFA: Best-Feature-Aware Fusion for Multi-View Fine-grained Manipulation

TL;DR

This paper introduces a plug-and-play best-feature-aware fusion strategy for multi-view manipulation tasks, improving success rates by selectively emphasizing the most relevant views during different manipulation stages.

Contribution

It proposes a lightweight, adaptable network to predict view importance scores, enabling effective multi-view feature fusion for fine-grained manipulation tasks.

Findings

Outperforms baselines with 22-46% higher success rate

Effectively identifies and emphasizes key views at different manipulation stages

Enhances manipulation success in various fine-grained tasks

Abstract

In real-world scenarios, multi-view cameras are typically employed for fine-grained manipulation tasks. Existing approaches (e.g., ACT) tend to treat multi-view features equally and directly concatenate them for policy learning. However, it will introduce redundant visual information and bring higher computational costs, leading to ineffective manipulation. For a fine-grained manipulation task, it tends to involve multiple stages while the most contributed view for different stages is varied over time. In this paper, we propose a plug-and-play best-feature-aware (BFA) fusion strategy for multi-view manipulation tasks, which is adaptable to various policies. Built upon the visual backbone of the policy network, we design a lightweight network to predict the importance score of each view. Based on the predicted importance scores, the reweighted multi-view features are subsequently fused and input into the end-to-end policy network, enabling seamless integration. Notably, our method demonstrates outstanding performance in fine-grained manipulations. Experimental results show that our approach outperforms multiple baselines by 22-46% success rate on different tasks. Our work provides new insights and inspiration for tackling key challenges in fine-grained manipulations.