MCGA: A Multi-task Classical Chinese Literary Genre Audio Corpus

TL;DR

The paper introduces MCGA, a comprehensive 119-hour audio corpus of Classical Chinese literary genres designed to evaluate and advance multimodal large language models in underexplored audio tasks.

Contribution

It presents a new multi-task audio corpus for Classical Chinese literature, evaluates existing models, and proposes domain-specific metrics to improve MLLMs in this niche.

Findings

Current MLLMs face significant challenges on MCGA tasks.

The corpus enables evaluation across six diverse tasks.

Proposed metrics help measure speech-text capability consistency.

Abstract

With the rapid advancement of Multimodal Large Language Models (MLLMs), their potential has gained significant attention in Chinese Classical Studies (CCS). While existing research primarily focuses on text and visual modalities, the audio corpus within this domain remains largely underexplored. To bridge this gap, we introduce the Multi-task Classical Chinese Literary Genre Audio Corpus (MCGA), a 119-hour corpus comprising 22,000 audio samples. It encompasses a diverse range of literary genres across six tasks: Automatic Speech Recognition (ASR), Speech-to-Text Translation (S2TT), Speech Emotion Captioning (SEC), Spoken Question Answering (SQA), Speech Understanding (SU), and Speech Reasoning (SR). Through the evaluation of ten MLLMs, our experimental results demonstrate that current MLLMs still face substantial challenges on the MCGA test set. Furthermore, we introduce a domain-specific metric for SEC and a metric to measure the consistency between speech and text capabilities. We release MCGA to the public to facilitate the development of more robust MLLMs. MCGA Corpus: https://github.com/yxduir/MCGA