Cross-modal Spectrum Transformation Network For Acoustic Scene classification

TL;DR

This paper proposes a novel spectrum transformation network that converts audio features into imagined visual features, improving acoustic scene classification, especially in unseen environments, by leveraging audio-visual relationships learned from videos.

Contribution

Introduces an acoustic spectrum transformation network that generates imagined visual features from log-mel spectra to enhance generalization in scene classification.

Findings

Outperforms traditional spectrum features on DCASE and ESC-50 datasets.

Shows improved generalization to unseen environments.

Utilizes large-scale YouTube videos for training.

Abstract

Convolutional neural networks (CNNs) with log-mel spectrum features have shown promising results for acoustic scene classification tasks. However, the performance of these CNN based classifiers is still lacking as they do not generalise well for unknown environments. To address this issue, we introduce an acoustic spectrum transformation network where traditional log-mel spectrums are transformed into imagined visual features (IVF). The imagined visual features are learned by exploiting the relationship between audio and visual features present in video recordings. An auto-encoder is used to encode images as visual features and a transformation network learns how to generate imagined visual features from log-mel. Our model is trained on a large dataset of Youtube videos. We test our proposed method on the scene classification task of DCASE and ESC-50, where our method outperforms other spectrum features, especially for unseen environments.