Audiovisual Moments in Time: A Large-Scale Annotated Dataset of Audiovisual Actions

TL;DR

This paper introduces AVMIT, a large-scale annotated audiovisual dataset with feature embeddings, demonstrating improved audiovisual event recognition performance using RNNs trained on audiovisual-specific data.

Contribution

The paper presents AVMIT, a new annotated dataset with feature embeddings, and shows that training RNNs on audiovisual-specific data enhances event recognition accuracy.

Findings

Training RNNs on AVMIT data improves accuracy by up to 5.94%.

Audiovisual-specific training outperforms modality-agnostic approaches.

AVMIT provides a valuable resource for audiovisual research.

Abstract

We present Audiovisual Moments in Time (AVMIT), a large-scale dataset of audiovisual action events. In an extensive annotation task 11 participants labelled a subset of 3-second audiovisual videos from the Moments in Time dataset (MIT). For each trial, participants assessed whether the labelled audiovisual action event was present and whether it was the most prominent feature of the video. The dataset includes the annotation of 57,177 audiovisual videos, each independently evaluated by 3 of 11 trained participants. From this initial collection, we created a curated test set of 16 distinct action classes, with 60 videos each (960 videos). We also offer 2 sets of pre-computed audiovisual feature embeddings, using VGGish/YamNet for audio data and VGG16/EfficientNetB0 for visual data, thereby lowering the barrier to entry for audiovisual DNN research. We explored the advantages of AVMIT annotations and feature embeddings to improve performance on audiovisual event recognition. A series of 6 Recurrent Neural Networks (RNNs) were trained on either AVMIT-filtered audiovisual events or modality-agnostic events from MIT, and then tested on our audiovisual test set. In all RNNs, top 1 accuracy was increased by 2.71-5.94\% by training exclusively on audiovisual events, even outweighing a three-fold increase in training data. We anticipate that the newly annotated AVMIT dataset will serve as a valuable resource for research and comparative experiments involving computational models and human participants, specifically when addressing research questions where audiovisual correspondence is of critical importance.