Attentional Graph Convolutional Network for Structure-aware Audio-Visual Scene Classification

TL;DR

This paper introduces an end-to-end attentional graph convolutional network (AGCN) that captures structure-aware features for audio-visual scene classification, emphasizing salient and semantic regions in both modalities.

Contribution

The paper proposes a novel AGCN framework that constructs and utilizes multiple graphs to represent salient and contextual information in audio-visual data for improved scene recognition.

Findings

Achieved promising results on multiple scene recognition datasets.

Effectively visualized graphs to highlight salient regions.

Demonstrated the importance of structure-aware features in audio-visual understanding.

Abstract

Audio-Visual scene understanding is a challenging problem due to the unstructured spatial-temporal relations that exist in the audio signals and spatial layouts of different objects and various texture patterns in the visual images. Recently, many studies have focused on abstracting features from convolutional neural networks while the learning of explicit semantically relevant frames of sound signals and visual images has been overlooked. To this end, we present an end-to-end framework, namely attentional graph convolutional network (AGCN), for structure-aware audio-visual scene representation. First, the spectrogram of sound and input image is processed by a backbone network for feature extraction. Then, to build multi-scale hierarchical information of input features, we utilize an attention fusion mechanism to aggregate features from multiple layers of the backbone network. Notably, to well represent the salient regions and contextual information of audio-visual inputs, the salient acoustic graph (SAG) and contextual acoustic graph (CAG), salient visual graph (SVG), and contextual visual graph (CVG) are constructed for the audio-visual scene representation. Finally, the constructed graphs pass through a graph convolutional network for structure-aware audio-visual scene recognition. Extensive experimental results on the audio, visual and audio-visual scene recognition datasets show that promising results have been achieved by the AGCN methods. Visualizing graphs on the spectrograms and images have been presented to show the effectiveness of proposed CAG/SAG and CVG/SVG that could focus on the salient and semantic relevant regions.