Tactile Beyond Pixels: Multisensory Touch Representations for Robot Manipulation
Carolina Higuera, Akash Sharma, Taosha Fan, Chaithanya Krishna Bodduluri, Byron Boots, Michael Kaess, Mike Lambeta, Tingfan Wu, Zixi Liu, Francois Robert Hogan, Mustafa Mukadam

TL;DR
This paper introduces Sparsh-X, a multisensory touch representation combining four tactile modalities, which enhances robot manipulation, imitation learning, and physical property inference by leveraging self-supervised learning on contact-rich data.
Contribution
Sparsh-X is the first multisensory touch representation across four modalities, improving manipulation success, robustness, and physical property inference in robotic tasks.
Findings
Boosts policy success rates by 63% in manipulation tasks.
Improves robustness in object state recovery by 90%.
Increases accuracy in physical property inference by 48%.
Abstract
We present Sparsh-X, the first multisensory touch representations across four tactile modalities: image, audio, motion, and pressure. Trained on ~1M contact-rich interactions collected with the Digit 360 sensor, Sparsh-X captures complementary touch signals at diverse temporal and spatial scales. By leveraging self-supervised learning, Sparsh-X fuses these modalities into a unified representation that captures physical properties useful for robot manipulation tasks. We study how to effectively integrate real-world touch representations for both imitation learning and tactile adaptation of sim-trained policies, showing that Sparsh-X boosts policy success rates by 63% over an end-to-end model using tactile images and improves robustness by 90% in recovering object states from touch. Finally, we benchmark Sparsh-X ability to make inferences about physical properties, such as object-action…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTactile and Sensory Interactions · Robot Manipulation and Learning · Interactive and Immersive Displays
