Lost in Speech: Benchmarking, Evaluation, and Parsing of Spoken Code-Switching Beyond Standard UD Assumptions

TL;DR

This paper addresses the challenges of syntactic parsing in spoken code-switching by introducing a new benchmark, an ambiguity-aware evaluation metric, and a novel parsing framework that significantly improves robustness and interpretability.

Contribution

It presents a linguistically grounded taxonomy, a new benchmark SpokeBench, the FLEX-UD evaluation metric, and the DECAP parsing framework for better spoken CSW analysis beyond standard UD assumptions.

Findings

Existing parsers perform poorly on spoken CSW data.

DECAP improves parsing robustness by up to 52.6%.

FLEX-UD reveals qualitative improvements masked by standard metrics.

Abstract

Spoken code-switching (CSW) challenges syntactic parsing in ways not observed in written text. Disfluencies, repetition, ellipsis, and discourse-driven structure routinely violate standard Universal Dependencies (UD) assumptions, causing parsers and large language models (LLMs) to fail despite strong performance on written data. These failures are compounded by rigid evaluation metrics that conflate genuine structural errors with acceptable variation. In this work, we present a systems-oriented approach to spoken CSW parsing. We introduce a linguistically grounded taxonomy of spoken CSW phenomena and SpokeBench, an expert-annotated gold benchmark designed to test spoken-language structure beyond standard UD assumptions. We further propose FLEX-UD, an ambiguity-aware evaluation metric, which reveals that existing parsing techniques perform poorly on spoken CSW by penalizing linguistically plausible analyses as errors. We then propose DECAP, a decoupled agentic parsing framework that isolates spoken-phenomena handling from core syntactic analysis. Experiments show that DECAP produces more robust and interpretable parses without retraining and achieves up to 52.6% improvements over existing parsing techniques. FLEX-UD evaluations further reveal qualitative improvements that are masked by standard metrics.