OmniUMI: Towards Physically Grounded Robot Learning via Human-Aligned Multimodal Interaction

TL;DR

OmniUMI introduces a multimodal robot learning framework that captures visual, tactile, and force data through a handheld system, enabling contact-rich manipulation tasks with human-aligned interaction.

Contribution

The paper presents OmniUMI, a unified system for physically grounded robot learning using multimodal sensing and natural human interaction, extending diffusion policies for contact-rich tasks.

Findings

Reliable sensing of contact and force signals demonstrated.

Strong performance in force-sensitive manipulation tasks.

Unified framework improves contact-rich manipulation capabilities.

Abstract

UMI-style interfaces enable scalable robot learning, but existing systems remain largely visuomotor, relying primarily on RGB observations and trajectory while providing only limited access to physical interaction signals. This becomes a fundamental limitation in contact-rich manipulation, where success depends on contact dynamics such as tactile interaction, internal grasping force, and external interaction wrench that are difficult to infer from vision alone. We present OmniUMI, a unified framework for physically grounded robot learning via human-aligned multimodal interaction. OmniUMI synchronously captures RGB, depth, trajectory, tactile sensing, internal grasping force, and external interaction wrench within a compact handheld system, while maintaining collection--deployment consistency through a shared embodiment design. To support human-aligned demonstration, OmniUMI enables natural perception and modulation of internal grasping force, external interaction wrench, and tactile interaction through bilateral gripper feedback and the handheld embodiment. Built on this interface, we extend diffusion policy with visual, tactile, and force-related observations, and deploy the learned policy through impedance-based execution for unified regulation of motion and contact behavior. Experiments demonstrate reliable sensing and strong downstream performance on force-sensitive pick-and-place, interactive surface erasing, and tactile-informed selective release. Overall, OmniUMI combines physically grounded multimodal data acquisition with human-aligned interaction, providing a scalable foundation for learning contact-rich manipulation.