Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models

TL;DR

This paper investigates enhancing audio language models for low-resource languages like Thai by optimizing data mixtures and integrating instruction-following capabilities, resulting in a model that outperforms existing open-source models and rivals state-of-the-art systems.

Contribution

It introduces Typhoon-Audio, a model that improves low-resource language processing and instruction-following by balancing multilingual and language-specific training data.

Findings

Typhoon-Audio outperforms existing open-source models.

The model achieves comparable performance to Gemini-1.5-Pro.

Balanced data mixtures enhance instruction-following in low-resource languages.

Abstract

Audio language models process audio inputs using textual prompts for tasks like speech recognition and audio captioning. Although built on multilingual pre-trained components, most are trained primarily on English, limiting their usability for other languages. This paper evaluates audio language models on Thai, a low-resource language, and finds that they lack emergent cross-lingual abilities despite their multilingual foundations. To address this, we explore data mixtures that optimize audio language models for both a target language and English while integrating audio comprehension and speech instruction-following into a unified model. Our experiments provide insights into improving instruction-following in low-resource languages by balancing language-specific and multilingual training data. The proposed model, Typhoon-Audio, significantly outperforms existing open-source models and achieves performance comparable to state-of-the-art Gemini-1.5-Pro in both English and Thai.