Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing

TL;DR

This paper introduces a flexible discrete multimodal language model that leverages pretrained large language models and mixed supervision to improve performance across various speech, text, and vision tasks.

Contribution

It presents a novel decoder-only DMLM that integrates multimodal data, explores training strategies, and demonstrates significant benefits from pretrained LLMs and codebook initialization.

Findings

DMLM improves across multiple tasks and datasets.

Pretraining with LLM enhances ASR performance.

Using Whisper-derived codebooks benefits the model.

Abstract

Recent work on discrete speech tokenization has paved the way for models that can seamlessly perform multiple tasks across modalities, e.g., speech recognition, text to speech, speech to speech translation. Moreover, large language models (LLMs) pretrained from vast text corpora contain rich linguistic information that can improve accuracy in a variety of tasks. In this paper, we present a decoder-only Discrete Multimodal Language Model (DMLM), which can be flexibly applied to multiple tasks (ASR, T2S, S2TT, etc.) and modalities (text, speech, vision). We explore several critical aspects of discrete multi-modal models, including the loss function, weight initialization, mixed training supervision, and codebook. Our results show that DMLM benefits significantly, across multiple tasks and datasets, from a combination of supervised and unsupervised training. Moreover, for ASR, it benefits from initializing DMLM from a pretrained LLM, and from a codebook derived from Whisper activations.