WhatsAI: Transforming Meta Ray-Bans into an Extensible Generative AI Platform for Accessibility

TL;DR

WhatsAI is an extensible, hackable framework that transforms Meta Ray-Bans into personalized, community-driven visual accessibility tools for blind and visually impaired users, enabling real-time scene understanding and object recognition.

Contribution

It introduces the first fully hackable template integrating Meta Ray-Bans with WhatsApp for accessible AI applications, fostering democratized innovation in visual accessibility technology.

Findings

Enables real-time scene description, object detection, and OCR for BVI users.

Provides a community-driven, extensible platform for visual accessibility innovations.

Facilitates integration with popular messaging apps for practical use cases.

Abstract

Multi-modal generative AI models integrated into wearable devices have shown significant promise in enhancing the accessibility of visual information for blind or visually impaired (BVI) individuals, as evidenced by the rapid uptake of Meta Ray-Bans among BVI users. However, the proprietary nature of these platforms hinders disability-led innovation of visual accessibility technologies. For instance, OpenAI showcased the potential of live, multi-modal AI as an accessibility resource in 2024, yet none of the presented applications have reached BVI users, despite the technology being available since then. To promote the democratization of visual access technology development, we introduce WhatsAI, a prototype extensible framework that empowers BVI enthusiasts to leverage Meta Ray-Bans to create personalized wearable visual accessibility technologies. Our system is the first to offer a fully hackable template that integrates with WhatsApp, facilitating robust Accessible Artificial Intelligence Implementations (AAII) that enable blind users to conduct essential visual assistance tasks, such as real-time scene description, object detection, and Optical Character Recognition (OCR), utilizing standard machine learning techniques and cutting-edge visual language models. The extensible nature of our framework aspires to cultivate a community-driven approach, led by BVI hackers and innovators to tackle the complex challenges associated with visual accessibility.