OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation

TL;DR

OmniVTA introduces a large-scale visuo-tactile dataset and a novel world-model framework that enhances contact-rich robotic manipulation by integrating predictive contact modeling with high-frequency tactile feedback.

Contribution

The paper presents OmniViTac, a comprehensive visuo-tactile dataset, and OmniVTA, a new world-model-based framework that improves manipulation performance through integrated contact dynamics modeling.

Findings

OmniVTA outperforms existing methods in contact-rich tasks.

The framework generalizes well to unseen objects and configurations.

Closed-loop tactile feedback improves manipulation accuracy.

Abstract

Contact-rich manipulation tasks, such as wiping and assembly, require accurate perception of contact forces, friction changes, and state transitions that cannot be reliably inferred from vision alone. Despite growing interest in visuo-tactile manipulation, progress is constrained by two persistent limitations: existing datasets are small in scale and narrow in task coverage, and current methods treat tactile signals as passive observations rather than using them to model contact dynamics or enable closed-loop control explicitly. In this paper, we present \textbf{OmniViTac}, a large-scale visuo-tactile-action dataset comprising $21{,}000+$ trajectories across $86$ tasks and $100+$ objects, organized into six physics-grounded interaction patterns. Building on this dataset, we propose \textbf{OmniVTA}, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules: a self-supervised tactile encoder, a two-stream visuo-tactile world model for predicting short-horizon contact evolution, a contact-aware fusion policy for action generation, and a 60Hz reflexive controller that corrects deviations between predicted and observed tactile signals in a closed loop. Real-robot experiments across all six interaction categories show that OmniVTA outperforms existing methods and generalizes well to unseen objects and geometric configurations, confirming the value of combining predictive contact modeling with high-frequency tactile feedback for contact-rich manipulation. All data, models, and code will be made publicly available on the project website at https://mrsecant.github.io/OmniVTA.