General In-Hand Object Rotation with Vision and Touch

TL;DR

This paper presents RotateIt, a system that uses multimodal vision and touch inputs, trained in simulation and distilled for real-world deployment, to enable precise in-hand object rotation along multiple axes.

Contribution

The paper introduces a novel visuotactile transformer-based system that fuses visual and tactile data for in-hand object rotation, trained in simulation and adapted for real-world noisy inputs.

Findings

Significant performance improvements over prior methods

Effective fusion of visual and tactile sensing

Successful deployment on realistic noisy sensory inputs

Abstract

We introduce RotateIt, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our system is trained in simulation, where it has access to ground-truth object shapes and physical properties. Then we distill it to operate on realistic yet noisy simulated visuotactile and proprioceptive sensory inputs. These multimodal inputs are fused via a visuotactile transformer, enabling online inference of object shapes and physical properties during deployment. We show significant performance improvements over prior methods and the importance of visual and tactile sensing.