Can Foundation Models Reliably Identify Spatial Hazards? A Case Study on Curb Segmentation

TL;DR

This study evaluates the performance of foundation models in curb segmentation for assistive navigation, revealing significant challenges and proposing solutions to improve accuracy and real-time applicability for visually impaired users.

Contribution

Introduces the largest curb segmentation dataset and benchmarks foundation models, highlighting their limitations and proposing filtering techniques to enhance segmentation accuracy.

Findings

High false-positive rates up to 95% in curb detection

Average inference time of 3.70 seconds per image

Foundation models require refinement for practical assistive use

Abstract

Curbs serve as vital borders that delineate safe pedestrian zones from potential vehicular traffic hazards. Curbs also represent a primary spatial hazard during dynamic navigation with significant stumbling potential. Such vulnerabilities are particularly exacerbated for persons with blindness and low vision (PBLV). Accurate visual-based discrimination of curbs is paramount for assistive technologies that aid PBLV with safe navigation in urban environments. Herein, we investigate the efficacy of curb segmentation for foundation models. We introduce the largest curb segmentation dataset to-date to benchmark leading foundation models. Our results show that state-of-the-art foundation models face significant challenges in curb segmentation. This is due to their high false-positive rates (up to 95%) with poor performance distinguishing curbs from curb-like objects or non-curb areas, such as sidewalks. In addition, the best-performing model averaged a 3.70-second inference time, underscoring problems in providing real-time assistance. In response, we propose solutions including filtered bounding box selections to achieve more accurate curb segmentation. Overall, despite the immediate flexibility of foundation models, their application for practical assistive technology applications still requires refinement. This research highlights the critical need for specialized datasets and tailored model training to address navigation challenges for PBLV and underscores implicit weaknesses in foundation models.