MIRAGE: Enabling Real-Time Automotive Mediated Reality

TL;DR

MIRAGE is an open-source tool that enables real-time automotive mediated reality, enhancing driving safety by overlaying and modifying visual information in real vehicles using advanced computational models.

Contribution

This work introduces MIRAGE, the first real-time AMR system for vehicles, implementing 15 effects with state-of-the-art models, and evaluates its usability and limitations in real-world driving scenarios.

Findings

Participants enjoyed the AMR experience and identified technical limitations.

MIRAGE successfully demonstrates real-time effects in on-road conditions.

The study provides insights into potential use cases and ethical considerations for AMR.

Abstract

Traffic is inherently dangerous, with around 1.19 million fatalities annually. Automotive Mediated Reality (AMR) can enhance driving safety by overlaying critical information (e.g., outlines, icons, text) on key objects to improve awareness, altering objects' appearance to simplify traffic situations, and diminishing their appearance to minimize distractions. However, real-world AMR evaluation remains limited due to technical challenges. To fill this sim-to-real gap, we present MIRAGE, an open-source tool that enables real-time AMR in real vehicles. MIRAGE implements 15 effects across the AMR spectrum of augmented, diminished, and modified reality using state-of-the-art computational models for object detection and segmentation, depth estimation, and inpainting. In an on-road expert user study (N=9) of MIRAGE, participants enjoyed the AMR experience while pointing out technical limitations and identifying use cases for AMR. We discuss these results in relation to prior work and outline implications for AMR ethics and interaction design.