MARS: Multimodal Active Robotic Sensing for Articulated Characterization

TL;DR

MARS is a multi-modal active sensing framework that improves articulated object perception for robots by combining RGB and point cloud data with reinforcement learning to optimize viewpoints, outperforming existing methods.

Contribution

The paper introduces MARS, a novel multi-modal fusion and active sensing framework that enhances articulated object characterization in robotic perception.

Findings

Outperforms state-of-the-art in joint parameter estimation

Reduces errors through active viewpoint optimization

Generalizes effectively to real-world objects

Abstract

Precise perception of articulated objects is vital for empowering service robots. Recent studies mainly focus on point cloud, a single-modal approach, often neglecting vital texture and lighting details and assuming ideal conditions like optimal viewpoints, unrepresentative of real-world scenarios. To address these limitations, we introduce MARS, a novel framework for articulated object characterization. It features a multi-modal fusion module utilizing multi-scale RGB features to enhance point cloud features, coupled with reinforcement learning-based active sensing for autonomous optimization of observation viewpoints. In experiments conducted with various articulated object instances from the PartNet-Mobility dataset, our method outperformed current state-of-the-art methods in joint parameter estimation accuracy. Additionally, through active sensing, MARS further reduces errors, demonstrating enhanced efficiency in handling suboptimal viewpoints. Furthermore, our method effectively generalizes to real-world articulated objects, enhancing robot interactions. Code is available at https://github.com/robhlzeng/MARS.