OpusLM: A Family of Open Unified Speech Language Models

TL;DR

OpusLM introduces a family of open, scalable speech language models trained on extensive speech-text data, achieving competitive performance across speech recognition, synthesis, and text tasks, with full transparency and open resources.

Contribution

This work presents the design, training, and evaluation of OpusLMs, a new open-source family of SpeechLMs with up to 7B parameters, emphasizing transparency and comprehensive training strategies.

Findings

OpusLMs achieve comparable or superior performance to existing SpeechLMs.

Model size scaling and data selection strategies significantly impact performance.

Open release of code, data, and models supports further research.

Abstract

This paper presents Open Unified Speech Language Models (OpusLMs), a family of open foundational speech language models (SpeechLMs) up to 7B. Initialized from decoder-only text language models, the OpusLMs are continuously pre-trained on 213K hours of speech-text pairs and 292B text-only tokens. We demonstrate our OpusLMs achieve comparable (or even superior) performance with existing SpeechLMs in speech recognition, speech synthesis, and text-only capabilities. Technically, this paper articulates our SpeechLM designs on tokenization, multi-stream language models, and multi-stage training strategies. We experimentally demonstrate the importance of model size scaling and the effect of annealing data selection. The OpusLMs are all built from publicly available materials and are fully transparent models. We release our code, data, checkpoints, and training logs to facilitate open SpeechLM research