SARL: Spatially-Aware Self-Supervised Representation Learning for Visuo-Tactile Perception

TL;DR

SARL introduces a spatially-aware self-supervised learning framework that enhances visuo-tactile perception by maintaining spatial structure in feature representations, significantly improving robotic manipulation tasks involving geometry and texture understanding.

Contribution

This work presents SARL, a novel SSL framework that incorporates map-level objectives to preserve spatial information in fused visual-tactile data, outperforming existing methods in manipulation tasks.

Findings

SARL achieves a 30% reduction in MAE on edge-pose regression.

SARL outperforms nine SSL baselines across six downstream tasks.

Structured spatial equivariance is key for effective visuo-tactile perception.

Abstract

Contact-rich robotic manipulation requires representations that encode local geometry. Vision provides global context but lacks direct measurements of properties such as texture and hardness, whereas touch supplies these cues. Modern visuo-tactile sensors capture both modalities in a single fused image, yielding intrinsically aligned inputs that are well suited to manipulation tasks requiring visual and tactile information. Most self-supervised learning (SSL) frameworks, however, compress feature maps into a global vector, discarding spatial structure and misaligning with the needs of manipulation. To address this, we propose SARL, a spatially-aware SSL framework that augments the Bootstrap Your Own Latent (BYOL) architecture with three map-level objectives, including Saliency Alignment (SAL), Patch-Prototype Distribution Alignment (PPDA), and Region Affinity Matching (RAM), to keep attentional focus, part composition, and geometric relations consistent across views. These losses act on intermediate feature maps, complementing the global objective. SARL consistently outperforms nine SSL baselines across six downstream tasks with fused visual-tactile data. On the geometry-sensitive edge-pose regression task, SARL achieves a Mean Absolute Error (MAE) of 0.3955, a 30% relative improvement over the next-best SSL method (0.5682 MAE) and approaching the supervised upper bound. These findings indicate that, for fused visual-tactile data, the most effective signal is structured spatial equivariance, in which features vary predictably with object geometry, which enables more capable robotic perception.