SeaLLMs-Audio: Large Audio-Language Models for Southeast Asia

TL;DR

SeaLLMs-Audio is a pioneering large audio-language model designed for Southeast Asian languages, supporting multilingual, multimodal, and multi-task audio understanding and interaction, with promising performance on regional tasks.

Contribution

The paper introduces SeaLLMs-Audio, the first large audio-language model tailored for Southeast Asian languages, with a new benchmark SeaBench-Audio for evaluation.

Findings

Achieves competitive performance on SEA language tasks

Supports multiple languages and modalities effectively

Enables diverse audio-centric applications

Abstract

We introduce SeaLLMs-Audio, the first large audio-language model (LALM) tailored for multiple Southeast Asian (SEA) languages-Indonesian (id), Thai (th), and Vietnamese (vi)-alongside English (en) and Chinese (zh). Trained on a large-scale audio corpus, SeaLLMs-Audio exhibits strong performance across diverse audio-centric tasks, spanning fine-grained audio understanding and voice-based interaction. Its key features include: 1) Multilingual: the model primarily supports 5 languages, namely Indonesian, Thai, Vietnamese, English, and Chinese; 2) Multimodal: the model accepts flexible input modalities, including audio only, text only, as well as audio with text; 3) Multi-task: the model supports a wide range of tasks, including audio analysis tasks such as Audio Captioning, Automatic Speech Recognition, Speech-to-Text Translation, Speech Emotion Recognition, Speech Question Answering, and Speech Summarization. It also enables voice-based dialogue, including answering factual, mathematical, and general knowledge queries. As a significant step towards advancing audio LLMs in Southeast Asia, we expect SeaLLMs-Audio to benefit both the regional research community and industry. To automate LALM evaluation for Southeast Asia, we introduce SeaBench-Audio, a benchmark spanning multiple tasks. Experiments show that SeaLLMs-Audio achieves competitive performance compared with other LALMs on SEA languages.