Synesthesia of Vehicles: Tactile Data Synthesis from Visual Inputs

TL;DR

This paper introduces a novel framework inspired by human synesthesia that predicts tactile excitations from visual inputs in autonomous vehicles, enhancing safety by enabling proactive tactile perception.

Contribution

It presents a cross-modal alignment method and a visual-tactile generative model using latent diffusion for unsupervised tactile data synthesis in AVs.

Findings

VTSyn outperforms existing models in multiple metrics

The system improves AV safety through better tactile perception

A new multi-modal dataset was collected for evaluation

Abstract

Autonomous vehicles (AVs) rely on multi-modal fusion for safety, but current visual and optical sensors fail to detect road-induced excitations which are critical for vehicles' dynamic control. Inspired by human synesthesia, we propose the Synesthesia of Vehicles (SoV), a novel framework to predict tactile excitations from visual inputs for autonomous vehicles. We develop a cross-modal spatiotemporal alignment method to address temporal and spatial disparities. Furthermore, a visual-tactile synesthetic (VTSyn) generative model using latent diffusion is proposed for unsupervised high-quality tactile data synthesis. A real-vehicle perception system collected a multi-modal dataset across diverse road and lighting conditions. Extensive experiments show that VTSyn outperforms existing models in temporal, frequency, and classification performance, enhancing AV safety through proactive tactile perception.