Hierarchical Audio-Visual-Proprioceptive Fusion for Precise Robotic Manipulation

TL;DR

This paper introduces a hierarchical multimodal fusion framework that integrates audio, vision, and proprioception for precise robotic manipulation, demonstrating improved performance especially when acoustic cues are informative.

Contribution

The paper proposes a novel hierarchical fusion approach that explicitly models cross-modal interactions and conditions visual and proprioceptive data on acoustic cues for robotic tasks.

Findings

Outperforms state-of-the-art multimodal fusion methods in real-world tasks.

Effectively leverages acoustic cues to improve manipulation accuracy.

Mutual information analysis clarifies the role of audio in multimodal perception.

Abstract

Existing robotic manipulation methods primarily rely on visual and proprioceptive observations, which may struggle to infer contact-related interaction states in partially observable real-world environments. Acoustic cues, by contrast, naturally encode rich interaction dynamics during contact, yet remain underexploited in current multimodal fusion literature. Most multimodal fusion approaches implicitly assume homogeneous roles across modalities, and thus design flat and symmetric fusion structures. However, this assumption is ill-suited for acoustic signals, which are inherently sparse and contact-driven. To achieve precise robotic manipulation through acoustic-informed perception, we propose a hierarchical representation fusion framework that progressively integrates audio, vision, and proprioception. Our approach first conditions visual and proprioceptive representations on acoustic cues, and then explicitly models higher-order cross-modal interactions to capture complementary dependencies among modalities. The fused representation is leveraged by a diffusion-based policy to directly generate continuous robot actions from multimodal observations. The combination of end-to-end learning and hierarchical fusion structure enables the policy to exploit task-relevant acoustic information while mitigating interference from less informative modalities. The proposed method has been evaluated on real-world robotic manipulation tasks, including liquid pouring and cabinet opening. Extensive experiment results demonstrate that our approach consistently outperforms state-of-the-art multimodal fusion frameworks, particularly in scenarios where acoustic cues provide task-relevant information not readily available from visual observations alone. Furthermore, a mutual information analysis is conducted to interpret the effect of audio cues in robotic manipulation via multimodal fusion.