Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation

TL;DR

This paper introduces MS-Bot, a stage-guided multi-sensory fusion method that dynamically adjusts sensory modality priorities based on task stages, improving robotic manipulation performance and interpretability.

Contribution

The paper presents a novel stage-aware fusion approach that mimics human sensory integration, enabling robots to adaptively prioritize senses during complex tasks.

Findings

Enhanced manipulation accuracy in pouring and peg insertion tasks.

More effective and explainable sensory fusion compared to existing methods.

Robust performance across different stages of task execution.

Abstract

Humans possess a remarkable talent for flexibly alternating to different senses when interacting with the environment. Picture a chef skillfully gauging the timing of ingredient additions and controlling the heat according to the colors, sounds, and aromas, seamlessly navigating through every stage of the complex cooking process. This ability is founded upon a thorough comprehension of task stages, as achieving the sub-goal within each stage can necessitate the utilization of different senses. In order to endow robots with similar ability, we incorporate the task stages divided by sub-goals into the imitation learning process to accordingly guide dynamic multi-sensory fusion. We propose MS-Bot, a stage-guided dynamic multi-sensory fusion method with coarse-to-fine stage understanding, which dynamically adjusts the priority of modalities based on the fine-grained state within the predicted current stage. We train a robot system equipped with visual, auditory, and tactile sensors to accomplish challenging robotic manipulation tasks: pouring and peg insertion with keyway. Experimental results indicate that our approach enables more effective and explainable dynamic fusion, aligning more closely with the human fusion process than existing methods.