OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset

TL;DR

OLKAVS is the largest publicly available Korean audio-visual speech dataset, enabling advanced multi-modal research with extensive multi-view and noisy environment data, along with baseline models for speech recognition and lip reading.

Contribution

The paper introduces OLKAVS, the largest Korean audio-visual speech dataset with multi-view recordings and noise variations, and provides baseline models for key speech tasks.

Findings

Multi-modal and multi-view training improves performance.

OLKAVS enables research in Korean speech and speaker recognition.

Baseline models demonstrate dataset's effectiveness.

Abstract

Inspired by humans comprehending speech in a multi-modal manner, various audio-visual datasets have been constructed. However, most existing datasets focus on English, induce dependencies with various prediction models during dataset preparation, and have only a small number of multi-view videos. To mitigate the limitations, we recently developed the Open Large-scale Korean Audio-Visual Speech (OLKAVS) dataset, which is the largest among publicly available audio-visual speech datasets. The dataset contains 1,150 hours of transcribed audio from 1,107 Korean speakers in a studio setup with nine different viewpoints and various noise situations. We also provide the pre-trained baseline models for two tasks, audio-visual speech recognition and lip reading. We conducted experiments based on the models to verify the effectiveness of multi-modal and multi-view training over uni-modal and frontal-view-only training. We expect the OLKAVS dataset to facilitate multi-modal research in broader areas such as Korean speech recognition, speaker recognition, pronunciation level classification, and mouth motion analysis.