Addressing Out-of-Label Hazard Detection in Dashcam Videos: Insights from the COOOL Challenge

TL;DR

This paper introduces a comprehensive hazard detection system for dashcam videos that combines anomaly detection, ensemble weak classifiers with privacy measures, and vision-language models for captioning, achieving top results in the COOOL challenge.

Contribution

It presents a novel integrated approach for hazard detection and captioning in dashcam footage, utilizing unsupervised learning, ensemble methods, differential privacy, and advanced vision-language models.

Findings

Achieved highest scores in the COOOL challenge for hazard detection and captioning.

Demonstrated robustness of the ensemble method with privacy enhancements.

Effectively detected driver reactions and hazardous objects in dashcam videos.

Abstract

This paper presents a novel approach for hazard analysis in dashcam footage, addressing the detection of driver reactions to hazards, the identification of hazardous objects, and the generation of descriptive captions. We first introduce a method for detecting driver reactions through speed and sound anomaly detection, leveraging unsupervised learning techniques. For hazard detection, we employ a set of heuristic rules as weak classifiers, which are combined using an ensemble method. This ensemble approach is further refined with differential privacy to mitigate overconfidence, ensuring robustness despite the lack of labeled data. Lastly, we use state-of-the-art vision-language models for hazard captioning, generating descriptive labels for the detected hazards. Our method achieved the highest scores in the Challenge on Out-of-Label in Autonomous Driving, demonstrating its effectiveness across all three tasks. Source codes are publicly available at https://github.com/ffyyytt/COOOL_2025.