Scene-Aware Vectorized Memory Multi-Agent Framework with Cross-Modal Differentiated Quantization VLMs for Visually Impaired Assistance

TL;DR

This paper introduces a novel, memory-efficient vision-language model and multi-agent system that significantly improve environmental perception for visually impaired individuals while reducing computational requirements.

Contribution

It presents a cross-modal differentiated quantization framework and a scene-aware vectorized memory multi-agent system, enhancing efficiency and integrated scene understanding in assistive technology.

Findings

Memory reduced from 38GB to 11.3GB with minimal performance loss.

Achieves 2.83-3.52s latency for initial speech output.

Maintains high accuracy on OCR-VQA and MMBench benchmarks.

Abstract

Visually impaired individuals face significant challenges in environmental perception. Traditional assistive technologies often lack adaptive intelligence, focusing on individual components rather than integrated systems. While Vision-Language Models (VLMs) offer a promising path to richer, integrated understanding, their deployment is severely limited by substantial computational requirements, demanding dozens of gigabytes of memory. To address these gaps in computational efficiency and integrated design, this study proposes a dual technological innovation framework: a cross-modal differentiated quantization framework for VLMs and a scene-aware vectorized memory multi-agent system. The quantization framework implements differentiated strategies, reducing memory from 38GB to 11.3GB. The multi-agent system uses vectorized memory and perception-memory-reasoning workflows to provide environmental information beyond the current view, achieving 2.83-3.52s latency to initial speech output. Experiments show the quantized 19B-parameter model only experiences a 2.05% performance drop on MMBench and maintains 63.7 accuracy on OCR-VQA (original: 64.9), outperforming smaller models with equivalent memory. This research advances computational efficiency and assistive technology, offering comprehensive assistance in scene perception, text recognition, and navigation.