Lost in Transcription: How Speech-to-Text Errors Derail Code Understanding

TL;DR

This paper presents a multilingual speech-driven framework for code understanding that addresses transcription errors and enhances performance in voice-based programming tools, especially for Indic languages.

Contribution

It introduces a novel multilingual speech-to-code system with LLM-based refinement, improving code understanding accuracy in voice interfaces for non-English languages.

Findings

LLM-guided refinement significantly improves transcription accuracy.

Transcription errors notably impact downstream code understanding tasks.

Multilingual voice interfaces require code-sensitive adaptations.

Abstract

Code understanding is a foundational capability in software engineering tools and developer workflows. However, most existing systems are designed for English-speaking users interacting via keyboards, which limits accessibility in multilingual and voice-first settings, particularly in regions like India. Voice-based interfaces offer a more inclusive modality, but spoken queries involving code present unique challenges due to the presence of non-standard English usage, domain-specific vocabulary, and custom identifiers such as variable and function names, often combined with code-mixed expressions. In this work, we develop a multilingual speech-driven framework for code understanding that accepts spoken queries in a user native language, transcribes them using Automatic Speech Recognition (ASR), applies code-aware ASR output refinement using Large Language Models (LLMs), and interfaces with code models to perform tasks such as code question answering and code retrieval through benchmarks such as CodeSearchNet, CoRNStack, and CodeQA. Focusing on four widely spoken Indic languages and English, we systematically characterize how transcription errors impact downstream task performance. We also identified key failure modes in ASR for code and demonstrated that LLM-guided refinement significantly improves performance across both transcription and code understanding stages. Our findings underscore the need for code-sensitive adaptations in speech interfaces and offer a practical solution for building robust, multilingual voice-driven programming tools.