Robustness Risk of Conversational Retrieval: Identifying and Mitigating Noise Sensitivity in Qwen3-Embedding Model

TL;DR

This paper empirically investigates the robustness vulnerabilities of Qwen3-embedding models in conversational retrieval, revealing noise sensitivity issues and proposing lightweight query prompting as a mitigation strategy.

Contribution

It identifies a specific robustness failure mode in Qwen3 models under conversational noise and demonstrates how query prompting can mitigate this issue.

Findings

Structured dialogue noise can dominate retrieval results without query prompting.

Qwen3 models are more susceptible to noise than earlier variants and other baselines.

Lightweight query prompting effectively suppresses noise intrusion and stabilizes rankings.

Abstract

We present an empirical study of embedding-based retrieval under realistic conversational settings, where queries are short, dialogue-like, and weakly specified, and retrieval corpora contain structured conversational artifacts. Focusing on Qwen3-embedding models, we identify a deployment-relevant robustness vulnerability: under conversational retrieval without query prompting, structured dialogue-style noise can become disproportionately retrievable and intrude into top-ranked results, despite being semantically uninformative. This failure mode emerges consistently across model scales, remains largely invisible under standard clean-query benchmarks, and is significantly more pronounced in Qwen3 than in earlier Qwen variants and other widely used dense retrieval baselines. We further show that lightweight query prompting qualitatively alters retrieval behavior, effectively suppressing noise intrusion and restoring ranking stability. Our findings highlight an underexplored robustness risk in conversational retrieval and underscore the importance of evaluation protocols that reflect the complexities of deployed systems.