MST-R: Multi-Stage Tuning for Retrieval Systems and Metric Evaluation

TL;DR

This paper introduces MST-R, a multi-stage tuning strategy for retrieval systems that enhances domain adaptation and retrieval performance, achieving top results on the RegNLP challenge leaderboard.

Contribution

The paper proposes a novel multi-stage tuning approach for retrieval systems, combining fine-tuning, hybrid retrieval, and selective cross-attention adaptation, with benchmarking on a new dataset.

Findings

Significant performance improvements on the RegNLP challenge dataset.

A trivial answering method can outperform complex models on certain metrics.

Insights into metric gaming and future research directions.

Abstract

Regulatory documents are rich in nuanced terminology and specialized semantics. FRAG systems: Frozen retrieval-augmented generators utilizing pre-trained (or, frozen) components face consequent challenges with both retriever and answering performance. We present a system that adapts the retriever performance to the target domain using a multi-stage tuning (MST) strategy. Our retrieval approach, called MST-R (a) first fine-tunes encoders used in vector stores using hard negative mining, (b) then uses a hybrid retriever, combining sparse and dense retrievers using reciprocal rank fusion, and then (c) adapts the cross-attention encoder by fine-tuning only the top-k retrieved results. We benchmark the system performance on the dataset released for the RIRAG challenge (as part of the RegNLP workshop at COLING 2025). We achieve significant performance gains obtaining a top rank on the RegNLP challenge leaderboard. We also show that a trivial answering approach games the RePASs metric outscoring all baselines and a pre-trained Llama model. Analyzing this anomaly, we present important takeaways for future research.