Exploring the Potential of Multi-Modal AI for Driving Hazard Prediction

TL;DR

This paper introduces a new dataset and problem formulation for predicting driving hazards from static dashcam images, emphasizing high-level reasoning about future accidents using multi-modal AI.

Contribution

It presents the DHPR dataset and a novel approach to hazard prediction based on visual abductive reasoning from single images.

Findings

Baseline methods show promising results but highlight challenges in hazard prediction.

The dataset enables research on reasoning about future events from static images.

Future work can improve accuracy and incorporate additional modalities.

Abstract

This paper addresses the problem of predicting hazards that drivers may encounter while driving a car. We formulate it as a task of anticipating impending accidents using a single input image captured by car dashcams. Unlike existing approaches to driving hazard prediction that rely on computational simulations or anomaly detection from videos, this study focuses on high-level inference from static images. The problem needs predicting and reasoning about future events based on uncertain observations, which falls under visual abductive reasoning. To enable research in this understudied area, a new dataset named the DHPR (Driving Hazard Prediction and Reasoning) dataset is created. The dataset consists of 15K dashcam images of street scenes, and each image is associated with a tuple containing car speed, a hypothesized hazard description, and visual entities present in the scene. These are annotated by human annotators, who identify risky scenes and provide descriptions of potential accidents that could occur a few seconds later. We present several baseline methods and evaluate their performance on our dataset, identifying remaining issues and discussing future directions. This study contributes to the field by introducing a novel problem formulation and dataset, enabling researchers to explore the potential of multi-modal AI for driving hazard prediction.