Towards Dexterous Embodied Manipulation via Deep Multi-Sensory Fusion and Sparse Expert Scaling

TL;DR

DeMUSE introduces a deep multimodal framework combining vision, force, and geometry data with sparse experts and a diffusion transformer to enhance dexterous embodied manipulation, achieving state-of-the-art success rates.

Contribution

The paper presents DeMUSE, a novel multi-sensory fusion framework with adaptive normalization and sparse expert scaling for improved physical interaction in manipulation tasks.

Findings

Achieves 83.2% success in simulation

Achieves 72.5% success in real-world trials

Demonstrates state-of-the-art performance in complex tasks

Abstract

Realizing dexterous embodied manipulation necessitates the deep integration of heterogeneous multimodal sensory inputs. However, current vision-centric paradigms often overlook the critical force and geometric feedback essential for complex tasks. This paper presents DeMUSE, a Deep Multimodal Unified Sparse Experts framework leveraging a Diffusion Transformer to integrate RGB, depth, and 6-axis force into a unified serialized stream. Adaptive Modality-specific Normalization (AdaMN) is employed to recalibrate modality-aware features, mitigating representation imbalance and harmonizing the heterogeneous distributions of multi-sensory signals. To facilitate efficient scaling, the architecture utilizes a Sparse Mixture-of-Experts (MoE) with shared experts, increasing model capacity for physical priors while maintaining the low inference latency required for real-time control. A Joint denoising objective synchronously synthesizes environmental evolution and action sequences to ensure physical consistency. Achieving success rates of 83.2% and 72.5% in simulation and real-world trials, DeMUSE demonstrates state-of-the-art performance, validating the necessity of deep multi-sensory integration for complex physical interactions.