LEALLA: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillation

TL;DR

This paper introduces LEALLA, a lightweight, language-agnostic sentence embedding model that uses knowledge distillation to achieve competitive performance across 109 languages while reducing inference overhead.

Contribution

The paper proposes a novel lightweight model architecture and distillation techniques for efficient, multilingual sentence embeddings, outperforming existing large models in speed and resource usage.

Findings

LEALLA achieves strong performance on multiple benchmarks.

The lightweight model significantly reduces inference time.

Knowledge distillation improves embedding quality.

Abstract

Large-scale language-agnostic sentence embedding models such as LaBSE (Feng et al., 2022) obtain state-of-the-art performance for parallel sentence alignment. However, these large-scale models can suffer from inference speed and computation overhead. This study systematically explores learning language-agnostic sentence embeddings with lightweight models. We demonstrate that a thin-deep encoder can construct robust low-dimensional sentence embeddings for 109 languages. With our proposed distillation methods, we achieve further improvements by incorporating knowledge from a teacher model. Empirical results on Tatoeba, United Nations, and BUCC show the effectiveness of our lightweight models. We release our lightweight language-agnostic sentence embedding models LEALLA on TensorFlow Hub.