Caraman at SemEval-2026 Task 8: Three-Stage Multi-Turn Retrieval with Query Rewriting, Hybrid Search, and Cross-Encoder Reranking

TL;DR

This paper presents a three-stage retrieval system for SemEval-2026 Task 8, combining query rewriting, hybrid search, and reranking, achieving competitive results across multiple domains.

Contribution

The authors introduce a novel multi-stage retrieval pipeline utilizing query rewriting with a fine-tuned language model, hybrid search, and cross-encoder reranking, demonstrating domain-specific tuning benefits.

Findings

Query rewriting with domain-specific temperature tuning improves retrieval performance.

Hybrid BM25 and dense retrieval with RRF outperforms baseline methods.

Complex strategies like multi-query expansion degrade performance in this setting.

Abstract

We describe our system for SemEval-2026 Task 8 (MTRAGEval), participating in Task A (Retrieval) across four English-language domains. Our approach employs a three-stage pipeline: (1) query rewriting via a LoRA-fine-tuned Qwen 2.5 7B model that transforms context-dependent follow-up questions into standalone queries, (2) hybrid BM25 and dense retrieval combined through Reciprocal Rank Fusion, and (3) cross-encoder reranking with BGE-reranker-v2-m3. On the official test set, the system achieves nDCG@5 of 0.531, ranking 8th out of 38 participating systems and 10.7% above the organizer baseline. Development comparisons reveal that domain-specific temperature tuning for query generation, where technical domains benefit from deterministic decoding and general domains from controlled randomness, provides consistent gains, while more complex strategies such as domain-aware prompting and multi-query expansion degrade performance.