Zero-shot Hazard Identification in Autonomous Driving: A Case Study on the COOOL Benchmark

TL;DR

This paper introduces a comprehensive approach for hazard detection in autonomous driving, combining multiple methods for detection, classification, and captioning, achieving high accuracy on the COOOL benchmark.

Contribution

The paper presents a novel multi-task pipeline integrating change point detection, object classification, and vision-language captioning for hazard identification in autonomous driving.

Findings

Outperformed baseline methods by 33% in error reduction.

Achieved 2nd place on the COOOL leaderboard.

Integrated diverse models for improved hazard detection and description.

Abstract

This paper presents our submission to the COOOL competition, a novel benchmark for detecting and classifying out-of-label hazards in autonomous driving. Our approach integrates diverse methods across three core tasks: (i) driver reaction detection, (ii) hazard object identification, and (iii) hazard captioning. We propose kernel-based change point detection on bounding boxes and optical flow dynamics for driver reaction detection to analyze motion patterns. For hazard identification, we combined a naive proximity-based strategy with object classification using a pre-trained ViT model. At last, for hazard captioning, we used the MOLMO vision-language model with tailored prompts to generate precise and context-aware descriptions of rare and low-resolution hazards. The proposed pipeline outperformed the baseline methods by a large margin, reducing the relative error by 33%, and scored 2nd on the final leaderboard consisting of 32 teams.