Audio-Visual Fusion Layers for Event Type Aware Video Recognition

TL;DR

This paper introduces a novel multi-task learning model with event-specific audio-visual fusion layers for improved video recognition, capturing complex multisensory interactions and revealing modality biases in datasets.

Contribution

It proposes event-specific fusion layers for multisensory integration, enabling better understanding of audio-visual relationships and multi-label outputs in video recognition tasks.

Findings

Event-specific layers discover unique audio-visual properties.

Model can output true multi-labels despite single-label training.

Framework exposes modality biases across datasets.

Abstract

Human brain is continuously inundated with the multisensory information and their complex interactions coming from the outside world at any given moment. Such information is automatically analyzed by binding or segregating in our brain. While this task might seem effortless for human brains, it is extremely challenging to build a machine that can perform similar tasks since complex interactions cannot be dealt with single type of integration but requires more sophisticated approaches. In this paper, we propose a new model to address the multisensory integration problem with individual event-specific layers in a multi-task learning scheme. Unlike previous works where single type of fusion is used, we design event-specific layers to deal with different audio-visual relationship tasks, enabling different ways of audio-visual formation. Experimental results show that our event-specific layers can discover unique properties of the audio-visual relationships in the videos. Moreover, although our network is formulated with single labels, it can output additional true multi-labels to represent the given videos. We demonstrate that our proposed framework also exposes the modality bias of the video data category-wise and dataset-wise manner in popular benchmark datasets.