WalkVLM:Aid Visually Impaired People Walking by Vision Language Model

TL;DR

This paper introduces WalkVLM, a vision-language model designed to assist visually impaired individuals by providing real-time walking guidance, supported by a new large-scale dataset and a benchmark for evaluating such systems.

Contribution

The paper presents the first large-scale walking assistance dataset and a novel WalkVLM model with hierarchical planning and temporal-aware prediction for improved guidance.

Findings

WalkVLM outperforms other VLMs in stream video processing tasks.

The dataset enables standardized training and evaluation for walking assistance.

WalkVLM generates concise, informative reminders effectively.

Abstract

Approximately 200 million individuals around the world suffer from varying degrees of visual impairment, making it crucial to leverage AI technology to offer walking assistance for these people. With the recent progress of vision-language models (VLMs), applying VLMs to offer walking guidance has become popular. However, the existing methods of walking guidance are mainly based on self-curated question-answering datasets that are not publicly accessible, without a standardized benchmark for training or evaluation. Moreover, walking assistance often requires real-time streaming video analysis and the generation of concise yet informative reminders, making VLMs struggle due to excessive responses and low efficiency in inferences. In this paper, we introduce the first large-scale dataset dedicated to walking assistance, comprising 12,000 video-annotation pairs, to provide a unified benchmark for training and evaluating systems to help visually-impaired individuals walk. Furthermore, a WalkVLM model is proposed, which employs chain of thought for hierarchical planning to generate concise but informative reminders and utilizes temporal-aware adaptive prediction to reduce the temporal redundancy of reminders. Finally, we have established a solid benchmark for blind walking task and verified the advantages of WalkVLM in stream video processing for this task compared to other VLMs. Our dataset and code are available at https://walkvlm2024.github.io.