MR.NAVI: Mixed-Reality Navigation Assistant for the Visually Impaired
Nicolas Pfitzer, Yifan Zhou, Marco Poggensee, Defne Kurtulus, Bessie Dominguez-Dager, Mihai Dusmanu, Marc Pollefeys, Zuria Bauer

TL;DR
MR.NAVI is a mixed-reality navigation system that uses computer vision and natural language processing to assist visually impaired users in unfamiliar environments, providing real-time scene understanding and navigation guidance.
Contribution
The paper introduces MR.NAVI, a novel mixed-reality system combining computer vision, NLP, and public transit integration for enhanced spatial awareness of visually impaired users.
Findings
Effective real-time scene understanding and navigation guidance.
Positive user study results demonstrating usability.
Successful obstacle avoidance and environment description.
Abstract
Over 43 million people worldwide live with severe visual impairment, facing significant challenges in navigating unfamiliar environments. We present MR.NAVI, a mixed reality system that enhances spatial awareness for visually impaired users through real-time scene understanding and intuitive audio feedback. Our system combines computer vision algorithms for object detection and depth estimation with natural language processing to provide contextual scene descriptions, proactive collision avoidance, and navigation instructions. The distributed architecture processes sensor data through MobileNet for object detection and employs RANSAC-based floor detection with DBSCAN clustering for obstacle avoidance. Integration with public transit APIs enables navigation with public transportation directions. Through our experiments with user studies, we evaluated both scene description and navigation…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTactile and Sensory Interactions · Multimodal Machine Learning Applications · Visual Attention and Saliency Detection
