Mind the Gap: Evaluating LLM Understanding of Human-Taught Road Safety Principles

TL;DR

This paper assesses the understanding of road safety principles by multi-modal large language models using a curated dataset, revealing significant gaps in safety reasoning and interpretation compared to human learning.

Contribution

It introduces a novel dataset of traffic signs and safety norms for evaluating LLMs and highlights their limitations in safety reasoning in a zero-shot setting.

Findings

Models struggle with safety reasoning tasks

Significant gaps exist between model and human understanding

Analysis suggests directions for improving LLM safety comprehension

Abstract

Following road safety norms is non-negotiable not only for humans but also for the AI systems that govern autonomous vehicles. In this work, we evaluate how well multi-modal large language models (LLMs) understand road safety concepts, specifically through schematic and illustrative representations. We curate a pilot dataset of images depicting traffic signs and road-safety norms sourced from school text books and use it to evaluate models capabilities in a zero-shot setting. Our preliminary results show that these models struggle with safety reasoning and reveal gaps between human learning and model interpretation. We further provide an analysis of these performance gaps for future research.