Is it safe to cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossing

TL;DR

This paper presents a novel approach using GPT-4V to interpret street crossing scenes, generate safety scores, and scene descriptions, aiding safe navigation for visually impaired individuals by leveraging multimodal models and real-world data.

Contribution

It introduces a multimodal model-based method for scene understanding and safety assessment at street crossings, advancing beyond traditional traffic signal recognition techniques.

Findings

LMM effectively predicts safety scores from egocentric images.

Scene descriptions generated support decision-making for visually impaired users.

Model demonstrates reasoning capabilities in complex street environments.

Abstract

Safely navigating street intersections is a complex challenge for blind and low-vision individuals, as it requires a nuanced understanding of the surrounding context - a task heavily reliant on visual cues. Traditional methods for assisting in this decision-making process often fall short, lacking the ability to provide a comprehensive scene analysis and safety level. This paper introduces an innovative approach that leverages large multimodal models (LMMs) to interpret complex street crossing scenes, offering a potential advancement over conventional traffic signal recognition techniques. By generating a safety score and scene description in natural language, our method supports safe decision-making for the blind and low-vision individuals. We collected crosswalk intersection data that contains multiview egocentric images captured by a quadruped robot and annotated the images with corresponding safety scores based on our predefined safety score categorization. Grounded on the visual knowledge, extracted from images, and text prompt, we evaluate a large multimodal model for safety score prediction and scene description. Our findings highlight the reasoning and safety score prediction capabilities of a LMM, activated by various prompts, as a pathway to developing a trustworthy system, crucial for applications requiring reliable decision-making support.