FaMTEB: Massive Text Embedding Benchmark in Persian Language

TL;DR

This paper presents FaMTEB, a comprehensive benchmark for Persian text embeddings, including new datasets, a novel task of summary retrieval, and chatbot evaluation datasets, to advance Persian NLP model evaluation.

Contribution

It introduces FaMTEB, the first extensive Persian text embedding benchmark with new datasets, a novel summary retrieval task, and chatbot evaluation datasets, along with an open-source platform.

Findings

Multiple Persian and multilingual models evaluated across diverse tasks.

Introduction of new Persian NLP datasets with no prior counterparts.

Benchmark and leaderboard available for future research.

Abstract

In this paper, we introduce a comprehensive benchmark for Persian (Farsi) text embeddings, built upon the Massive Text Embedding Benchmark (MTEB). Our benchmark includes 63 datasets spanning seven different tasks: classification, clustering, pair classification, reranking, retrieval, summary retrieval, and semantic textual similarity. The datasets are formed as a combination of existing, translated, and newly generated data, offering a diverse evaluation framework for Persian language models. Given the increasing use of text embedding models in chatbots, evaluation datasets are becoming inseparable ingredients in chatbot challenges and Retrieval-Augmented Generation systems. As a contribution, we include chatbot evaluation datasets in the MTEB benchmark for the first time. In addition, in this paper, we introduce the new task of summary retrieval which is not part of the tasks included in standard MTEB. Another contribution of this paper is the introduction of a substantial number of new Persian language NLP datasets suitable for training and evaluation, some of which have no previous counterparts in Persian. We evaluate the performance of several Persian and multilingual embedding models in a range of tasks. This work introduces an open-source benchmark with datasets, code and a public leaderboard.