V-HOP: Visuo-Haptic 6D Object Pose Tracking

TL;DR

V-HOP introduces a transformer-based visuo-haptic object pose tracker that effectively combines visual and tactile data, achieving superior robustness and generalization in real-world manipulation tasks compared to existing methods.

Contribution

The paper presents a novel unified haptic representation and a transformer-based visuo-haptic tracking framework that improves pose estimation across diverse sensors and embodiments.

Findings

Significant performance improvements on challenging sequences.

Enhanced generalization across different sensors and objects.

Outperforms state-of-the-art visual trackers in real-world experiments.

Abstract

Humans naturally integrate vision and haptics for robust object perception during manipulation. The loss of either modality significantly degrades performance. Inspired by this multisensory integration, prior object pose estimation research has attempted to combine visual and haptic/tactile feedback. Although these works demonstrate improvements in controlled environments or synthetic datasets, they often underperform vision-only approaches in real-world settings due to poor generalization across diverse grippers, sensor layouts, or sim-to-real environments. Furthermore, they typically estimate the object pose for each frame independently, resulting in less coherent tracking over sequences in real-world deployments. To address these limitations, we introduce a novel unified haptic representation that effectively handles multiple gripper embodiments. Building on this representation, we introduce a new visuo-haptic transformer-based object pose tracker that seamlessly integrates visual and haptic input. We validate our framework in our dataset and the Feelsight dataset, demonstrating significant performance improvement on challenging sequences. Notably, our method achieves superior generalization and robustness across novel embodiments, objects, and sensor types (both taxel-based and vision-based tactile sensors). In real-world experiments, we demonstrate that our approach outperforms state-of-the-art visual trackers by a large margin. We further show that we can achieve precise manipulation tasks by incorporating our real-time object tracking result into motion plans, underscoring the advantages of visuo-haptic perception. Project website: https://ivl.cs.brown.edu/research/v-hop