AudioRepInceptionNeXt: A lightweight single-stream architecture for efficient audio recognition

TL;DR

AudioRepInceptionNeXt is a lightweight, efficient neural network architecture for audio recognition that reduces computational costs by over 50% and speeds up inference by 1.28 times while maintaining accuracy, suitable for edge devices.

Contribution

The paper introduces AudioRepInceptionNeXt, a novel single-stream architecture with cascaded multi-scale depth-wise convolutions, inspired by efficient vision models, optimized for audio recognition tasks.

Findings

Reduces parameters and computations by over 50%.

Improves inference speed by 1.28 times over state-of-the-art CNNs.

Maintains comparable accuracy across various audio tasks.

Abstract

Recent research has successfully adapted vision-based convolutional neural network (CNN) architectures for audio recognition tasks using Mel-Spectrograms. However, these CNNs have high computational costs and memory requirements, limiting their deployment on low-end edge devices. Motivated by the success of efficient vision models like InceptionNeXt and ConvNeXt, we propose AudioRepInceptionNeXt, a single-stream architecture. Its basic building block breaks down the parallel multi-branch depth-wise convolutions with descending scales of k x k kernels into a cascade of two multi-branch depth-wise convolutions. The first multi-branch consists of parallel multi-scale 1 x k depth-wise convolutional layers followed by a similar multi-branch employing parallel multi-scale k x 1 depth-wise convolutional layers. This reduces computational and memory footprint while separating time and frequency processing of Mel-Spectrograms. The large kernels capture global frequencies and long activities, while small kernels get local frequencies and short activities. We also reparameterize the multi-branch design during inference to further boost speed without losing accuracy. Experiments show that AudioRepInceptionNeXt reduces parameters and computations by 50%+ and improves inference speed 1.28x over state-of-the-art CNNs like the Slow-Fast while maintaining comparable accuracy. It also learns robustly across a variety of audio recognition tasks. Codes are available at https://github.com/StevenLauHKHK/AudioRepInceptionNeXt.