SEA-BED: How Do Embedding Models Represent Southeast Asian Languages?

TL;DR

This paper introduces SEA-BED, a comprehensive benchmark for Southeast Asian languages, revealing that multilingual embedding models perform inconsistently across languages and tasks, challenging assumptions of stable semantic representations.

Contribution

The paper presents SEA-BED, a large-scale benchmark for SEA languages, and systematically evaluates embedding models, highlighting their uneven performance across languages and tasks.

Findings

No single model performs well across all SEA languages

Task difficulty varies significantly within languages

Performance varies across language-task combinations

Abstract

Multilingual text embeddings are often assumed to encode meaning in a perspective-independent semantic space, yielding stable similarity judgments across tasks and languages. Our results show that this assumption does not hold in practice. We introduce SEA-BED, a large-scale benchmark covering 10 Southeast Asian (SEA) languages and diverse embedding tasks, designed to systematically examine how embedding performance varies across tasks, languages, and language-task combinations. Across extensive evaluations, we observe that no single model performs uniformly well across SEA languages; task difficulty differs markedly within languages, and success on one task does not reliably generalize to others. Language-task analyses further reveal highly non-uniform performance landscapes, where performance varies across different language-task combinations. These findings call for closer attention to performance measurements that provide an expansive view across languages and tasks to uncover inconsistencies in semantic representation. Based on these observations, we provide insights for future model development, including data, algorithmic, and architectural considerations.