StreetReaderAI: Making Street View Accessible Using Context-Aware Multimodal AI

TL;DR

StreetReaderAI is a pioneering accessible street view tool that leverages multimodal AI to enable blind users to virtually explore and navigate streetscapes across the globe, enhancing accessibility and independence.

Contribution

It introduces the first accessible street view system combining context-aware multimodal AI, accessible controls, and conversational speech, specifically designed for blind users.

Findings

Supports POI investigations for blind users

Enables remote route planning

Validated with eleven blind users

Abstract

Interactive streetscape mapping tools such as Google Street View (GSV) and Meta Mapillary enable users to virtually navigate and experience real-world environments via immersive 360{\deg} imagery but remain fundamentally inaccessible to blind users. We introduce StreetReaderAI, the first-ever accessible street view tool, which combines context-aware, multimodal AI, accessible navigation controls, and conversational speech. With StreetReaderAI, blind users can virtually examine destinations, engage in open-world exploration, or virtually tour any of the over 220 billion images and 100+ countries where GSV is deployed. We iteratively designed StreetReaderAI with a mixed-visual ability team and performed an evaluation with eleven blind users. Our findings demonstrate the value of an accessible street view in supporting POI investigations and remote route planning. We close by enumerating key guidelines for future work.