A study on joint modeling and data augmentation of multi-modalities for audio-visual scene classification

TL;DR

This paper introduces joint modeling and data augmentation techniques to enhance audio-visual scene classification, leveraging pre-trained image models and novel mixup strategies, achieving state-of-the-art accuracy on a benchmark dataset.

Contribution

It presents a novel combination of joint modeling and data augmentation methods, including a new audio-visual mixup scheme, to improve AVSC performance.

Findings

Achieved 94.2% accuracy on TAU Urban Audio Visual Scenes 2021

Validated effectiveness of RandAugment operations for video data

Demonstrated benefits of audio-visual joint mixup in classification

Abstract

In this paper, we propose two techniques, namely joint modeling and data augmentation, to improve system performances for audio-visual scene classification (AVSC). We employ pre-trained networks trained only on image data sets to extract video embedding; whereas for audio embedding models, we decide to train them from scratch. We explore different neural network architectures for joint modeling to effectively combine the video and audio modalities. Moreover, data augmentation strategies are investigated to increase audio-visual training set size. For the video modality the effectiveness of several operations in RandAugment is verified. An audio-video joint mixup scheme is proposed to further improve AVSC performances. Evaluated on the development set of TAU Urban Audio Visual Scenes 2021, our final system can achieve the best accuracy of 94.2% among all single AVSC systems submitted to DCASE 2021 Task 1b.