Visualizing the Invisible: A Generative AR System for Intuitive Multi-Modal Sensor Data Presentation

TL;DR

Vivar is a novel AR system that visualizes multi-modal sensor data in 3D space using cross-modal embeddings and foundation models, making sensor data interpretation more intuitive for non-experts.

Contribution

The paper introduces Vivar, a system that integrates multi-modal sensor data into AR visualizations using a new cross-modal embedding approach and foundation models, reducing latency and enhancing intuitiveness.

Findings

11x faster content generation with caching strategies

Effective visualization of multi-modal sensor data in AR

User study confirms improved understanding and usability

Abstract

Understanding sensor data can be difficult for non-experts because of the complexity and different semantic meanings of sensor modalities. This leads to a need for intuitive and effective methods to present sensor information. However, creating intuitive sensor data visualizations presents three key challenges: the variability of sensor readings, gaps in domain comprehension, and the dynamic nature of sensor data. To address these issues, we propose Vivar, a novel system that integrates multi-modal sensor data and presents 3D volumetric content for AR visualization. In particular, we introduce a cross-modal embedding approach that maps sensor data into a pre-trained visual embedding space through barycentric interpolation. This approach accurately reflects value changes in multi-modal sensor information, ensuring that sensor variations are properly shown in visualization outcomes. Vivar also incorporates sensor-aware AR scene generation using foundation models and 3D Gaussian Splatting (3DGS) without requiring domain expertise. In addition, Vivar leverages latent reuse and caching strategies to accelerate 2D and AR content generation, demonstrating 11x latency reduction without compromising quality. A user study involving over 503 participants, including domain experts, demonstrates Vivar's effectiveness in accuracy, consistency, and real-world applicability, paving the way for more intuitive sensor data visualization.