Enabling American Sign Language Communication Under Low Data Rates

TL;DR

VC4ASL is a novel system that enables American Sign Language communication over audio channels in low-bandwidth video conferencing by encoding pose information and employing error correction, improving accessibility in degraded network conditions.

Contribution

This work introduces VC4ASL, a system that transmits pose-based ASL over audio channels, with novel error correction exploiting human pose structure, without requiring platform modifications.

Findings

Effective ASL communication over audio in low-bandwidth scenarios

Successful integration with existing video conferencing platforms

User studies confirm improved comprehension among ASL users

Abstract

In recent years, video conferencing applications have become increasingly prevalent, relying heavily on high-speed internet connectivity. When such connectivity is lacking, users often default to audio-only communication, a mode that significantly disadvantages American Sign Language (ASL) users, whose communication relies on hand gestures, body movement, and facial expressions. In this work, we introduce VC4ASL, a system designed to enable ASL communication over the audio channel of existing video conferencing applications, even in the absence of reliable video. VC4ASL integrates seamlessly with current platforms without requiring any modifications. Our approach establishes a communication channel through audio by encoding and transmitting human pose information, which is then rendered to reconstruct signed content. We propose novel receive-side error detection and correction mechanisms that exploit the inherent structural constraints of human pose data. To evaluate the system, we simulate network-degraded environments, generate pose-based ASL video sequences, and conduct user studies to assess comprehension among ASL users. Experimental results demonstrate that VC4ASL effectively facilitates intelligible ASL communication over audio in low-bandwidth scenarios where video transmission is impaired.