Speech Enhancement Using Continuous Embeddings of Neural Audio Codec

TL;DR

This paper introduces a novel speech enhancement method that leverages continuous embeddings from a pretrained Neural Audio Codec, achieving high efficiency and low complexity suitable for cloud-based audio transmission.

Contribution

The work presents a new NAC-based speech enhancement approach using continuous embeddings, reducing computational complexity and maintaining competitive performance.

Findings

Achieves real-time factor of 0.005, significantly faster than baselines.

Reduces model complexity by 18-fold compared to Sepformer.

Performs comparably to larger dataset-trained baselines.

Abstract

Recent advancements in Neural Audio Codec (NAC) models have inspired their use in various speech processing tasks, including speech enhancement (SE). In this work, we propose a novel, efficient SE approach by leveraging the pre-quantization output of a pretrained NAC encoder. Unlike prior NAC-based SE methods, which process discrete speech tokens using Language Models (LMs), we perform SE within the continuous embedding space of the pretrained NAC, which is highly compressed along the time dimension for efficient representation. Our lightweight SE model, optimized through an embedding-level loss, delivers results comparable to SE baselines trained on larger datasets, with a significantly lower real-time factor of 0.005. Additionally, our method achieves a low GMAC of 3.94, reducing complexity 18-fold compared to Sepformer in a simulated cloud-based audio transmission environment. This work highlights a new, efficient NAC-based SE solution, particularly suitable for cloud applications where NAC is used to compress audio before transmission. Copyright 20XX IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.