Ti-Audio: The First Multi-Dialectal End-to-End Speech LLM for Tibetan

TL;DR

Ti-Audio is a pioneering multi-dialectal end-to-end Speech-LLM for Tibetan, leveraging cross-dialectal cooperation and novel alignment techniques to excel in low-resource, dialect-diverse environments.

Contribution

The paper introduces Ti-Audio, the first multi-dialectal Tibetan Speech-LLM, with a Dynamic Q-Former Adapter and a mutual assistance strategy for low-resource dialectal speech tasks.

Findings

Achieves state-of-the-art results on Tibetan speech benchmarks.

Effectively utilizes dialectal mutual assistance to improve performance.

Validates cross-dialectal cooperation as a scalable approach.

Abstract

Recent advances in Speech Large Language Models (Speech-LLMs) have made significant progress, greatly enhancing multimodal interaction capabilities.However, their application in low-resource and dialect-diverse environments still faces challenges. The severe scarcity of Tibetan data, coupled with the phonetic differences among its major dialects (\"U-Tsang, Amdo, and Kham), is a prime example of this challenge. This paper proposes Ti-Audio, the first multi-dialectal end-to-end Speech-LLM for Tibetan. To efficiently align speech and text, we introduce a Dynamic Q-Former Adapter that extracts essential acoustic features from variable-length speech, ensuring stable cross-modal alignment even with limited data. At the data level, we leverage mutual assistance among related dialects to alleviate data scarcity and employ a temperature-based sampling strategy to maximize this synergy. Experimental results demonstrate that Ti-Audio achieves state-of-the-art performance on Tibetan benchmarks for automatic speech recognition and speech translation. Our work validates the effectiveness of cross-dialectal cooperation and provides a scalable paradigm for the development of Speech-LLM in low-resource scenarios.