MINERS: Multilingual Language Models as Semantic Retrievers

TL;DR

This paper introduces MINERS, a benchmark for evaluating multilingual language models' ability to perform semantic retrieval across many languages, including low-resource and complex language settings, showing competitive results without fine-tuning.

Contribution

The paper presents a comprehensive benchmark and framework for assessing multilingual LMs' semantic retrieval capabilities across over 200 languages, including challenging scenarios.

Findings

Retrieving semantically similar embeddings achieves competitive performance.

Multilingual LMs can effectively perform semantic retrieval without fine-tuning.

The benchmark covers over 200 diverse languages, including low-resource and code-switching contexts.

Abstract

Words have been represented in a high-dimensional vector space that encodes their semantic similarities, enabling downstream applications such as retrieving synonyms, antonyms, and relevant contexts. However, despite recent advances in multilingual language models (LMs), the effectiveness of these models' representations in semantic retrieval contexts has not been comprehensively explored. To fill this gap, this paper introduces the MINERS, a benchmark designed to evaluate the ability of multilingual LMs in semantic retrieval tasks, including bitext mining and classification via retrieval-augmented contexts. We create a comprehensive framework to assess the robustness of LMs in retrieving samples across over 200 diverse languages, including extremely low-resource languages in challenging cross-lingual and code-switching settings. Our results demonstrate that by solely retrieving semantically similar embeddings yields performance competitive with state-of-the-art approaches, without requiring any fine-tuning.