A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids

TL;DR

This paper presents a pioneering transceiver prototype for cloud-based audio-visual speech enhancement tailored for multimodal hearing aids, addressing high data rate, low latency, and real-time processing challenges.

Contribution

It introduces novel uplink and downlink frame structures optimized for AV speech enhancement in hearing aids, with real-time implementation and evaluation.

Findings

Supports multiple data rates for uplink channels

Achieves low latency suitable for real-time AV processing

Demonstrates feasibility with software defined radio hardware

Abstract

In this paper, we design a first of its kind transceiver (PHY layer) prototype for cloud-based audio-visual (AV) speech enhancement (SE) complying with high data rate and low latency requirements of future multimodal hearing assistive technology. The innovative design needs to meet multiple challenging constraints including up/down link communications, delay of transmission and signal processing, and real-time AV SE models processing. The transceiver includes device detection, frame detection, frequency offset estimation, and channel estimation capabilities. We develop both uplink (hearing aid to the cloud) and downlink (cloud to hearing aid) frame structures based on the data rate and latency requirements. Due to the varying nature of uplink information (audio and lip-reading), the uplink channel supports multiple data rate frame structure, while the downlink channel has a fixed data rate frame structure. In addition, we evaluate the latency of different PHY layer blocks of the transceiver for developed frame structures using LabVIEW NXG. This can be used with software defined radio (such as Universal Software Radio Peripheral) for real-time demonstration scenarios.