Do We Need Domain-Specific Embedding Models? An Empirical Investigation

TL;DR

This paper empirically investigates whether general-purpose embedding models are sufficient for domain-specific tasks, using finance as a case study, and introduces FinMTEB to evaluate domain-specific embedding performance.

Contribution

The paper introduces FinMTEB, a finance domain-specific benchmark, and provides evidence that current general-purpose models underperform in specialized domains, highlighting the need for domain-specific embeddings.

Findings

Significant performance drop of models on FinMTEB compared to MTEB

Performance on MTEB does not correlate with FinMTEB results

Domain-specific models are necessary for capturing specialized linguistic patterns

Abstract

Embedding models play a crucial role in representing and retrieving information across various NLP applications. Recent advancements in Large Language Models (LLMs) have further enhanced the performance of embedding models, which are trained on massive amounts of text covering almost every domain. These models are often benchmarked on general-purpose datasets like Massive Text Embedding Benchmark (MTEB), where they demonstrate superior performance. However, a critical question arises: Is the development of domain-specific embedding models necessary when general-purpose models are trained on vast corpora that already include specialized domain texts? In this paper, we empirically investigate this question, choosing the finance domain as an example. We introduce the Finance Massive Text Embedding Benchmark (FinMTEB), a counterpart to MTEB that consists of financial domain-specific text datasets. We evaluate the performance of seven state-of-the-art embedding models on FinMTEB and observe a significant performance drop compared to their performance on MTEB. To account for the possibility that this drop is driven by FinMTEB's higher complexity, we propose four measures to quantify dataset complexity and control for this factor in our analysis. Our analysis provides compelling evidence that state-of-the-art embedding models struggle to capture domain-specific linguistic and semantic patterns. Moreover, we find that the performance of general-purpose embedding models on MTEB is not correlated with their performance on FinMTEB, indicating the need for domain-specific embedding benchmarks for domain-specific embedding models. This study sheds light on developing domain-specific embedding models in the LLM era. FinMTEB comes with open-source code at https://github.com/yixuantt/FinMTEB