A Multimodal Assistive System for Product Localization and Retrieval for People who are Blind or have Low Vision

TL;DR

This paper introduces a multimodal wearable assistive system that combines object detection, vision-language models, and auditory guidance to help people who are blind or have low vision locate and retrieve products independently in shopping environments.

Contribution

The paper presents a novel multimodal system integrating detection, navigation, and correction modules specifically designed for assistive shopping for pBLV users.

Findings

Product detection accuracy near 100% at close range

Navigation accuracy up to 94.4% with vision-language models

Correction accuracy exceeds 86% under optimal conditions

Abstract

Shopping is a routine activity for sighted individuals, yet for people who are blind or have low vision (pBLV), locating and retrieving products in physical environments remains a challenge. This paper presents a multimodal wearable assistive system that integrates object detection with vision-language models to support independent product or item retrieval, with the goal of enhancing users'autonomy and sense of agency. The system operates through three phases: product search, which identifies target products using YOLO-World detection combined with embedding similarity and color histogram matching; product navigation, which provides spatialized sonification and VLM-generated verbal descriptions to guide users toward the target; and product correction, which verifies whether the user has reached the correct product and provides corrective feedback when necessary. Technical evaluation demonstrated promising performance across all modules, with product detection achieving near-perfect accuracy at close range and high accuracy when facing shelves within 1.5 m. VLM-based navigation achieved up to 94.4% accuracy, and correction accuracy exceeded 86% under optimal model configurations. These results demonstrate the system's potential to address the last-meter problem in assistive shopping. Future work will focus on user studies with pBLV participants and integration with multi-scale navigation ecosystems.