ALAS: Measuring Latent Speech-Text Alignment For Spoken Language Understanding In Multimodal LLMs

TL;DR

This paper introduces ALAS, a novel metric for quantifying the alignment between audio and text representations in multimodal LLMs, addressing a gap in evaluation methods for Spoken Language Understanding.

Contribution

ALAS provides a standardized way to measure latent speech-text alignment in multimodal LLMs, enabling better assessment of model performance across tasks.

Findings

ALAS correlates with task performance in SLU tasks.

ALAS reveals meaningful layer-wise alignment patterns.

The metric is effective across different multimodal tasks.

Abstract

Large Language Models (LLMs) are increasingly used in Spoken Language Understanding (SLU), where effective multimodal learning depends on the alignment between audio and text. Despite various fusion methods, no standard metric exists to assess this alignment. This work introduces ALAS (Automatic Latent Alignment Score), a metric that evaluates alignment by measuring correlations between audio and text representations across transformer layers. Experiments on Spoken Question Answering and Emotion Recognition show that ALAS captures meaningful patterns across tasks and layers.