The Multimodal Information Based Speech Processing (MISP) 2023 Challenge: Audio-Visual Target Speaker Extraction

TL;DR

The MISP 2023 challenge introduces a novel audio-visual target speaker extraction task aimed at improving speech recognition accuracy in real-world, complex acoustic environments, setting a new benchmark in multimodal speech processing.

Contribution

This paper presents the first benchmark for AVTSE in real-world scenarios, including task setup, dataset, baseline system, and analysis of challenges in multimodal speech processing.

Findings

The task is highly demanding for current systems.

Baseline results highlight the difficulty of AVTSE in real environments.

The challenge encourages innovative solutions for robust speech extraction.

Abstract

Previous Multimodal Information based Speech Processing (MISP) challenges mainly focused on audio-visual speech recognition (AVSR) with commendable success. However, the most advanced back-end recognition systems often hit performance limits due to the complex acoustic environments. This has prompted a shift in focus towards the Audio-Visual Target Speaker Extraction (AVTSE) task for the MISP 2023 challenge in ICASSP 2024 Signal Processing Grand Challenges. Unlike existing audio-visual speech enhance-ment challenges primarily focused on simulation data, the MISP 2023 challenge uniquely explores how front-end speech processing, combined with visual clues, impacts back-end tasks in real-world scenarios. This pioneering effort aims to set the first benchmark for the AVTSE task, offering fresh insights into enhancing the ac-curacy of back-end speech recognition systems through AVTSE in challenging and real acoustic environments. This paper delivers a thorough overview of the task setting, dataset, and baseline system of the MISP 2023 challenge. It also includes an in-depth analysis of the challenges participants may encounter. The experimental results highlight the demanding nature of this task, and we look forward to the innovative solutions participants will bring forward.