OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation

TL;DR

OmniFlatten is an end-to-end GPT-based model designed for real-time, full-duplex spoken dialogue, effectively modeling natural conversation behaviors with low latency through a multi-stage training scheme and modality flattening.

Contribution

The paper introduces a novel multi-stage training process and a flattening operation to adapt GPT models for natural, low-latency full-duplex spoken dialogue without architecture modifications.

Findings

Effective modeling of complex conversational behaviors

Low-latency real-time speech and text generation

Unified training approach across modalities

Abstract

Full-duplex spoken dialogue systems significantly surpass traditional turn-based dialogue systems, as they allow simultaneous bidirectional communication, closely mirroring human-human interactions. However, achieving low latency and natural interactions in full-duplex dialogue systems remains a significant challenge, especially considering human conversation dynamics such as interruptions, backchannels, and overlapping speech. In this paper, we introduce a novel End-to-End GPT-based model OmniFlatten for full-duplex conversation, capable of effectively modeling the complex behaviors inherent to natural conversations with low latency. To achieve full-duplex conversation capabilities, we propose a multi-stage post-training scheme that progressively adapts a text large language model (LLM) backbone into a speech-text dialogue LLM, capable of generating text and speech in real time, without modifying the architecture of the backbone LLM. The training process comprises three stages: modality alignment, half-duplex dialogue learning, and full-duplex dialogue learning. In all training stages, we standardize the data using a flattening operation, which enables unifying the training methods and the GPT backbone across different modalities and tasks. Our approach offers a simple modeling technique and a promising research direction for developing efficient and natural end-to-end full-duplex spoken dialogue systems. Audio samples of dialogues generated by OmniFlatten can be found at this web site (https://omniflatten.github.io/).