MTEB-French: Resources for French Sentence Embedding Evaluation and Analysis

TL;DR

This paper introduces MTEB-French, a comprehensive benchmark for evaluating French sentence embeddings, including datasets, models, and analysis, to advance NLP tasks in French.

Contribution

It extends the MTEB benchmark to French, providing new datasets, a large-scale comparison of models, and analysis of factors influencing performance.

Findings

Large multilingual models pre-trained on sentence similarity perform well.

No single model dominates across all tasks.

Open-source datasets and leaderboard facilitate future research.

Abstract

Recently, numerous embedding models have been made available and widely used for various NLP tasks. The Massive Text Embedding Benchmark (MTEB) has primarily simplified the process of choosing a model that performs well for several tasks in English, but extensions to other languages remain challenging. This is why we expand MTEB to propose the first massive benchmark of sentence embeddings for French. We gather 15 existing datasets in an easy-to-use interface and create three new French datasets for a global evaluation of 8 task categories. We compare 51 carefully selected embedding models on a large scale, conduct comprehensive statistical tests, and analyze the correlation between model performance and many of their characteristics. We find out that even if no model is the best on all tasks, large multilingual models pre-trained on sentence similarity perform exceptionally well. Our work comes with open-source code, new datasets and a public leaderboard.