Human-in-the-loop Reasoning For Traffic Sign Detection: Collaborative Approach Yolo With Video-llava

TL;DR

This paper introduces a collaborative human-in-the-loop approach combining YOLO and Video-LLava to improve traffic sign detection accuracy in adverse weather and challenging visual conditions for autonomous vehicles.

Contribution

It presents a novel method integrating video analysis, reasoning, and human guidance to enhance YOLO's traffic sign detection in semi-real-world scenarios.

Findings

Improved detection accuracy under adverse weather conditions

Effective collaboration between YOLO and Video-LLava demonstrated

Enhanced recognition of speed limit signs in challenging environments

Abstract

Traffic Sign Recognition (TSR) detection is a crucial component of autonomous vehicles. While You Only Look Once (YOLO) is a popular real-time object detection algorithm, factors like training data quality and adverse weather conditions (e.g., heavy rain) can lead to detection failures. These failures can be particularly dangerous when visual similarities between objects exist, such as mistaking a 30 km/h sign for a higher speed limit sign. This paper proposes a method that combines video analysis and reasoning, prompting with a human-in-the-loop guide large vision model to improve YOLOs accuracy in detecting road speed limit signs, especially in semi-real-world conditions. It is hypothesized that the guided prompting and reasoning abilities of Video-LLava can enhance YOLOs traffic sign detection capabilities. This hypothesis is supported by an evaluation based on human-annotated accuracy metrics within a dataset of recorded videos from the CARLA car simulator. The results demonstrate that a collaborative approach combining YOLO with Video-LLava and reasoning can effectively address challenging situations such as heavy rain and overcast conditions that hinder YOLOs detection capabilities.