Visionary Co-Driver: Enhancing Driver Perception of Potential Risks with LLM and HUD

TL;DR

This paper presents Visionary Co-Driver, a system that uses large language models and HUD to improve driver perception of non-collision risks by analyzing roadside behavior and alerting drivers through eye movement-based cues.

Contribution

It introduces a novel system combining LLMs and video processing to detect non-collision risks and dynamically inform drivers via HUD, enhancing situational awareness.

Findings

User study shows improved risk perception with Visionary Co-Driver.

System effectively identifies risky road users in non-collision scenarios.

Drivers' attention is successfully guided to potential hazards.

Abstract

Drivers' perception of risky situations has always been a challenge in driving. Existing risk-detection methods excel at identifying collisions but face challenges in assessing the behavior of road users in non-collision situations. This paper introduces Visionary Co-Driver, a system that leverages large language models to identify non-collision roadside risks and alert drivers based on their eye movements. Specifically, the system combines video processing algorithms and LLMs to identify potentially risky road users. These risks are dynamically indicated on an adaptive heads-up display interface to enhance drivers' attention. A user study with 41 drivers confirms that Visionary Co-Driver improves drivers' risk perception and supports their recognition of roadside risks.