ATEB: Evaluating and Improving Advanced NLP Tasks for Text Embedding Models

TL;DR

This paper introduces ATEB, a new benchmark for evaluating advanced NLP tasks like safety and factuality in text embeddings, and proposes a retrieval-based reformulation method that improves performance on these tasks.

Contribution

The paper presents a novel benchmark for assessing complex NLP capabilities and a retrieval reformulation approach that enhances embedding models' understanding of safety and factuality.

Findings

Benchmark reveals gaps in current embedding models' advanced capabilities.

Retrieval reformulation improves factuality classification by 8%.

Retrieval reformulation improves safety classification by 13%.

Abstract

Traditional text embedding benchmarks primarily evaluate embedding models' capabilities to capture semantic similarity. However, more advanced NLP tasks require a deeper understanding of text, such as safety and factuality. These tasks demand an ability to comprehend and process complex information, often involving the handling of sensitive content, or the verification of factual statements against reliable sources. We introduce a new benchmark designed to assess and highlight the limitations of embedding models trained on existing information retrieval data mixtures on advanced capabilities, which include factuality, safety, instruction following, reasoning and document-level understanding. This benchmark includes a diverse set of tasks that simulate real-world scenarios where these capabilities are critical and leads to identification of the gaps of the currently advanced embedding models. Furthermore, we propose a novel method that reformulates these various tasks as retrieval tasks. By framing tasks like safety or factuality classification as retrieval problems, we leverage the strengths of retrieval models in capturing semantic relationships while also pushing them to develop a deeper understanding of context and content. Using this approach with single-task fine-tuning, we achieved performance gains of 8\% on factuality classification and 13\% on safety classification. Our code and data will be publicly available.