SASST: Leveraging Syntax-Aware Chunking and LLMs for Simultaneous Speech Translation

TL;DR

This paper introduces SASST, a novel speech translation framework that uses syntax-aware chunking and large language models to improve translation quality and timing in multilingual settings.

Contribution

It presents a grammar-based chunking strategy combined with an end-to-end LLM-based architecture for simultaneous speech translation, enhancing coherence and translation accuracy.

Findings

Significant translation quality improvements across multiple languages.

Effective integration of syntax-aware chunking with LLMs for SimulST.

Validation on CoVoST2 corpus demonstrates robustness and effectiveness.

Abstract

This work proposes a grammar-based chunking strategy that segments input streams into semantically complete units by parsing dependency relations (e.g., noun phrase boundaries, verb-object structures) and punctuation features. The method ensures chunk coherence and minimizes semantic fragmentation. Building on this mechanism, we present SASST (Syntax-Aware Simultaneous Speech Translation), an end-to-end framework integrating frozen Whisper encoder and decoder-only LLM. The unified architecture dynamically outputs translation tokens or <WAIT> symbols to jointly optimize translation timing and content, with target-side reordering addressing word-order divergence. Experiments on CoVoST2 multilingual corpus En-{De, Zh, Ja} demonstrate significant translation quality improvements across languages and validate the effectiveness of syntactic structures in LLM-driven SimulST systems.