Beyond Modality Limitations: A Unified MLLM Approach to Automated Speaking Assessment with Effective Curriculum Learning

TL;DR

This paper explores the use of Multimodal Large Language Models for comprehensive automated speaking assessment, introducing a curriculum learning strategy to improve delivery evaluation and outperform existing methods.

Contribution

It is the first systematic study of MLLM for ASA, proposing Speech-First Multimodal Training to enhance speech modeling and assessment accuracy.

Findings

MLLM-based systems improve holistic assessment PCC from 0.783 to 0.846.

SFMT enhances delivery aspect evaluation with a 4% accuracy gain.

The approach offers a new avenue for multimodal automated speaking assessment.

Abstract

Traditional Automated Speaking Assessment (ASA) systems exhibit inherent modality limitations: text-based approaches lack acoustic information while audio-based methods miss semantic context. Multimodal Large Language Models (MLLM) offer unprecedented opportunities for comprehensive ASA by simultaneously processing audio and text within unified frameworks. This paper presents a very first systematic study of MLLM for comprehensive ASA, demonstrating the superior performance of MLLM across the aspects of content and language use . However, assessment on the delivery aspect reveals unique challenges, which is deemed to require specialized training strategies. We thus propose Speech-First Multimodal Training (SFMT), leveraging a curriculum learning principle to establish more robust modeling foundations of speech before cross-modal synergetic fusion. A series of experiments on a benchmark dataset show MLLM-based systems can elevate the holistic assessment performance from a PCC value of 0.783 to 0.846. In particular, SFMT excels in the evaluation of the delivery aspect, achieving an absolute accuracy improvement of 4% over conventional training approaches, which also paves a new avenue for ASA.