A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions

TL;DR

This study systematically evaluates large vision-language models for 2D object detection in automated driving, revealing their strengths in semantic robustness and potential as safety validators under adverse conditions.

Contribution

It provides the first comprehensive comparison of LVLMs against classical detectors like YOLO for safety-critical perception in SOTIF scenarios.

Findings

LVLMs outperform YOLO in recall under natural, complex scenarios.

LVLMs show greater robustness to environmental degradations.

YOLO maintains higher geometric precision in synthetic perturbations.

Abstract

Reliable environmental perception remains one of the main obstacles for safe operation of automated vehicles. Safety of the Intended Functionality (SOTIF) concerns safety risks from perception insufficiencies, particularly under adverse conditions where conventional detectors often falter. While Large Vision-Language Models (LVLMs) demonstrate promising semantic reasoning, their quantitative effectiveness for safety-critical 2D object detection is underexplored. This paper presents a systematic evaluation of ten representative LVLMs using the PeSOTIF dataset, a benchmark specifically curated for long-tail traffic scenarios and environmental degradations. Performance is quantitatively compared against the classical perception approach, a YOLO-based detector. Experimental results reveal a critical trade-off: top-performing LVLMs (e.g., Gemini 3, Doubao) surpass the YOLO baseline in recall by over 25% in complex natural scenarios, exhibiting superior robustness to visual degradation. Conversely, the baseline retains an advantage in geometric precision for synthetic perturbations. These findings highlight the complementary strengths of semantic reasoning versus geometric regression, supporting the use of LVLMs as high-level safety validators in SOTIF-oriented automated driving systems.