Continuous Speech Tokens Makes LLMs Robust Multi-Modality Learners

TL;DR

This paper introduces Flow-Omni, a continuous speech token model that enhances multi-modality learning in LLMs, improving robustness and real-time speech interaction by replacing discrete tokens with continuous representations.

Contribution

The paper proposes a novel continuous speech token approach using flow matching loss, enabling more robust multi-modality learning in LLMs compared to traditional discrete token methods.

Findings

Continuous speech tokens improve robustness over discrete tokens.

Flow-Omni achieves low latency real-time speech interaction.

Continuous tokens mitigate representation loss issues.

Abstract

Recent advances in GPT-4o like multi-modality models have demonstrated remarkable progress for direct speech-to-speech conversation, with real-time speech interaction experience and strong speech understanding ability. However, current research focuses on discrete speech tokens to align with discrete text tokens for language modelling, which depends on an audio codec with residual connections or independent group tokens, such a codec usually leverages large scale and diverse datasets training to ensure that the discrete speech codes have good representation for varied domain, noise, style data reconstruction as well as a well-designed codec quantizer and encoder-decoder architecture for discrete token language modelling. This paper introduces Flow-Omni, a continuous speech token based GPT-4o like model, capable of real-time speech interaction and low streaming latency. Specifically, first, instead of cross-entropy loss only, we combine flow matching loss with a pretrained autoregressive LLM and a small MLP network to predict the probability distribution of the continuous-valued speech tokens from speech prompt. second, we incorporated the continuous speech tokens to Flow-Omni multi-modality training, thereby achieving robust speech-to-speech performance with discrete text tokens and continuous speech tokens together. Experiments demonstrate that, compared to discrete text and speech multi-modality training and its variants, the continuous speech tokens mitigate robustness issues by avoiding the inherent flaws of discrete speech code's representation loss for LLM.