Short-Term Memory Convolutions

TL;DR

This paper introduces Short-Term Memory Convolutions (STMC), a novel CNN-based method that significantly reduces inference latency and memory use in real-time time series processing, especially for audio applications.

Contribution

The paper presents STMC, a new convolutional approach that achieves low latency comparable to LSTMs, with faster training and stable convergence, applicable to speech separation and acoustic scene classification.

Findings

5-fold reduction in inference time for speech separation

Up to 4 times faster inference in acoustic scene classification

No loss in output quality or accuracy

Abstract

The real-time processing of time series signals is a critical issue for many real-life applications. The idea of real-time processing is especially important in audio domain as the human perception of sound is sensitive to any kind of disturbance in perceived signals, especially the lag between auditory and visual modalities. The rise of deep learning (DL) models complicated the landscape of signal processing. Although they often have superior quality compared to standard DSP methods, this advantage is diminished by higher latency. In this work we propose novel method for minimization of inference time latency and memory consumption, called Short-Term Memory Convolution (STMC) and its transposed counterpart. The main advantage of STMC is the low latency comparable to long short-term memory (LSTM) networks. Furthermore, the training of STMC-based models is faster and more stable as the method is based solely on convolutional neural networks (CNNs). In this study we demonstrate an application of this solution to a U-Net model for a speech separation task and GhostNet model in acoustic scene classification (ASC) task. In case of speech separation we achieved a 5-fold reduction in inference time and a 2-fold reduction in latency without affecting the output quality. The inference time for ASC task was up to 4 times faster while preserving the original accuracy.