"I Can See Forever!": Evaluating Real-time VideoLLMs for Assisting Individuals with Visual Impairments

TL;DR

This paper evaluates real-time VideoLLMs for assisting visually impaired individuals, introducing a new benchmark and dataset, and demonstrating improved hazard perception through fine-tuning models.

Contribution

It is the first comprehensive evaluation of VideoLLMs in real-world assistive scenarios for the visually impaired, including new datasets and fine-tuning methods.

Findings

GPT-4o achieves highest task success rate

Fine-tuning VITA-1.5 improves hazard recognition from 25% to 76%

User study highlights concerns about hazard perception

Abstract

The visually impaired population faces significant challenges in daily activities. While prior works employ vision language models for assistance, most focus on static content and cannot address real-time perception needs in complex environments. Recent VideoLLMs enable real-time vision and speech interaction, offering promising potential for assistive tasks. In this work, we conduct the first study evaluating their effectiveness in supporting daily life for visually impaired individuals. We first conducted a user survey with visually impaired participants to design the benchmark VisAssistDaily for daily life evaluation. Using VisAssistDaily, we evaluate popular VideoLLMs and find GPT-4o achieves the highest task success rate. We further conduct a user study to reveal concerns about hazard perception. To address this, we propose SafeVid, an environment-awareness dataset, and fine-tune VITA-1.5, improving risk recognition accuracy from 25.00% to 76.00%.We hope this work provides valuable insights and inspiration for future research in this field.