CleanS2S: Single-file Framework for Proactive Speech-to-Speech Interaction

TL;DR

CleanS2S introduces a unified, real-time speech-to-speech interaction framework with proactive dialogue capabilities, enabling more human-like and flexible conversational AI through novel memory and action judgment modules.

Contribution

It presents a single-file, transparent framework that integrates speech recognition, large language models, and proactive response strategies for conversational AI.

Findings

Achieves low latency with full-duplex websocket connections.

Supports five human-like response strategies.

Provides an extensible, transparent implementation for research.

Abstract

CleanS2S is a framework for human-like speech-to-speech interaction that advances conversational AI through single-file implementation and proactive dialogue capabilities. Our system integrates automatic speech recognition, large language models, and text-to-speech synthesis into a unified pipeline with real-time interruption handling, achieving low transition latency through full-duplex websocket connections and non-blocking I/O. Beyond conventional chatbot paradigms, we pioneer a proactive interaction mechanism, which combines memory systems with Subjective Action Judgement module, enabling five human-like response strategies: interruption, refusal, deflection, silence, and standard response. The memory module dynamically aggregates historical, and contextual data to inform interaction decisions. This approach breaks the rigid turn-based convention by allowing system-initiated dialog control and context-aware response selection. And we propose Action Judgement SFT that assesses input streams for responses strategies. The framework's single-file implementation with atomic configurations offers researchers unprecedented transparency and extensibility for interaction agents. The code of CleanS2S is released at \https://github.com/opendilab/CleanS2S.