WARDEN: Endangered Indigenous Language Transcription and Translation with 6 Hours of Training Data

TL;DR

This paper presents WARDEN, a specialized two-stage model for transcribing and translating the endangered Wardaman language with only 6 hours of data, outperforming larger models in low-resource settings.

Contribution

The paper introduces a novel two-stage approach for low-resource language transcription and translation, utilizing phonemic initialization and domain-specific knowledge.

Findings

WARDEN outperforms larger models with only 6 hours of data.

Phonemic initialization accelerates transcription model fine-tuning.

Domain-specific dictionary improves translation accuracy.

Abstract

This paper introduces WARDEN, an early language model system capable of transcribing and translating Wardaman, an endangered Australian indigenous language into English. The significant challenge we face is the lack of large-scale training data: in fact, we only have 6 hours of annotated audio. Therefore, while it is common practice to train a single model for transcription and translation using large datasets (like English to French), this practice is no longer viable in the Wardaman to English context. To tackle the low-resource challenge, we design WARDEN to have separate transcription and translation models: WARDEN first turns a Wardaman audio input into phonemic transcription, and then the transcription into English translation. Further, we propose two useful techniques to enhance performance. For transcription, we initialize the Wardaman token from Sundanese, a language that shares similar phonemes with Wardaman, to accelerate fine-tuning of the transcription model. For translation, we compile a Wardaman-English dictionary from expert annotations, and provide this domain-specific knowledge to a large language model (LLM) to reason and decide the final output. We empirically demonstrate that this two-stage design works better than data-hungry unified approaches in extremely low data settings. Using a mere 6 hours of annotated data, WARDEN outperforms larger open-source and proprietary models and establishes a strong baseline. Data and code are available.