Sanvaad: A Multimodal Accessibility Framework for ISL Recognition and Voice-Based Interaction

TL;DR

Sanvaad is a lightweight, multimodal framework that enables real-time, two-way communication for deaf and visually impaired users by integrating sign language recognition, voice translation, and speech interfaces on edge devices.

Contribution

It introduces a unified multimodal accessibility system combining efficient sign language recognition and voice-based interaction for inclusive communication.

Findings

Real-time ISL recognition on edge devices

Multilingual speech-to-text and text-to-speech integration

Support for both deaf and visually impaired users

Abstract

Communication between deaf users, visually im paired users, and the general hearing population often relies on tools that support only one direction of interaction. To address this limitation, this work presents Sanvaad, a lightweight multimodal accessibility framework designed to support real time, two-way communication. For deaf users, Sanvaad includes an ISL recognition module built on MediaPipe landmarks. MediaPipe is chosen primarily for its efficiency and low computational load, enabling the system to run smoothly on edge devices without requiring dedicated hardware. Spoken input from a phone can also be translated into sign representations through a voice-to-sign component that maps detected speech to predefined phrases and produces corresponding GIFs or alphabet-based visualizations. For visually impaired users, the framework provides a screen free voice interface that integrates multilingual speech recognition, text summarization, and text-to-speech generation. These components work together through a Streamlit-based interface, making the system usable on both desktop and mobile environments. Overall, Sanvaad aims to offer a practical and accessible pathway for inclusive communication by combining lightweight computer vision and speech processing tools within a unified framework.