Tragic Talkers: A Shakespearean Sound- and Light-Field Dataset for Audio-Visual Machine Learning Research

TL;DR

The paper introduces 'Tragic Talkers', a comprehensive 3D audio-visual dataset with synchronized multi-view video, spatial audio, and detailed annotations, aimed at advancing immersive audio-visual machine learning research.

Contribution

It provides a novel, high-quality dataset combining light-field video and spatial audio with extensive annotations for diverse talking scenarios, filling a critical gap in existing resources.

Findings

Dataset includes 30 sequences from 22 viewpoints and two microphone arrays.

Provides detailed annotations like face bounding boxes, pose keypoints, and dialogue transcriptions.

Enables research in object-based media, spatial audio, and multi-view analysis.

Abstract

3D audio-visual production aims to deliver immersive and interactive experiences to the consumer. Yet, faithfully reproducing real-world 3D scenes remains a challenging task. This is partly due to the lack of available datasets enabling audio-visual research in this direction. In most of the existing multi-view datasets, the accompanying audio is neglected. Similarly, datasets for spatial audio research primarily offer unimodal content, and when visual data is included, the quality is far from meeting the standard production needs. We present "Tragic Talkers", an audio-visual dataset consisting of excerpts from the "Romeo and Juliet" drama captured with microphone arrays and multiple co-located cameras for light-field video. Tragic Talkers provides ideal content for object-based media (OBM) production. It is designed to cover various conventional talking scenarios, such as monologues, two-people conversations, and interactions with considerable movement and occlusion, yielding 30 sequences captured from a total of 22 different points of view and two 16-element microphone arrays. Additionally, we provide voice activity labels, 2D face bounding boxes for each camera view, 2D pose detection keypoints, 3D tracking data of the mouth of the actors, and dialogue transcriptions. We believe the community will benefit from this dataset as it can assist multidisciplinary research. Possible uses of the dataset are discussed.