TMac: Temporal Multi-Modal Graph Learning for Acoustic Event Classification

TL;DR

TMac introduces a graph-based approach to model temporal relationships in multi-modal audiovisual data, significantly improving acoustic event classification by capturing dynamic intra- and inter-modal information.

Contribution

The paper presents a novel graph learning method that explicitly models temporal relations in multi-modal data for acoustic event classification, outperforming existing methods.

Findings

TMac achieves superior performance over state-of-the-art models.

Explicit temporal modeling enhances multi-modal acoustic event classification.

The approach effectively captures intra- and inter-modal temporal dynamics.

Abstract

Audiovisual data is everywhere in this digital age, which raises higher requirements for the deep learning models developed on them. To well handle the information of the multi-modal data is the key to a better audiovisual modal. We observe that these audiovisual data naturally have temporal attributes, such as the time information for each frame in the video. More concretely, such data is inherently multi-modal according to both audio and visual cues, which proceed in a strict chronological order. It indicates that temporal information is important in multi-modal acoustic event modeling for both intra- and inter-modal. However, existing methods deal with each modal feature independently and simply fuse them together, which neglects the mining of temporal relation and thus leads to sub-optimal performance. With this motivation, we propose a Temporal Multi-modal graph learning method for Acoustic event Classification, called TMac, by modeling such temporal information via graph learning techniques. In particular, we construct a temporal graph for each acoustic event, dividing its audio data and video data into multiple segments. Each segment can be considered as a node, and the temporal relationships between nodes can be considered as timestamps on their edges. In this case, we can smoothly capture the dynamic information in intra-modal and inter-modal. Several experiments are conducted to demonstrate TMac outperforms other SOTA models in performance. Our code is available at https://github.com/MGitHubL/TMac.