Multilingual Sentence-Level Semantic Search using Meta-Distillation Learning

TL;DR

This paper introduces MAML-Align, a meta-distillation approach for low-resource multilingual semantic search, which effectively transfers knowledge across languages and improves performance over traditional fine-tuning methods.

Contribution

It is the first to extend meta-distillation to multilingual semantic search, leveraging MAML-based learning to enhance cross-lingual transfer and generalization.

Findings

Meta-distillation boosts performance over fine-tuning.

Approach outperforms baseline models in multilingual settings.

Improves generalization to unseen languages.

Abstract

Multilingual semantic search is the task of retrieving relevant contents to a query expressed in different language combinations. This requires a better semantic understanding of the user's intent and its contextual meaning. Multilingual semantic search is less explored and more challenging than its monolingual or bilingual counterparts, due to the lack of multilingual parallel resources for this task and the need to circumvent "language bias". In this work, we propose an alignment approach: MAML-Align, specifically for low-resource scenarios. Our approach leverages meta-distillation learning based on MAML, an optimization-based Model-Agnostic Meta-Learner. MAML-Align distills knowledge from a Teacher meta-transfer model T-MAML, specialized in transferring from monolingual to bilingual semantic search, to a Student model S-MAML, which meta-transfers from bilingual to multilingual semantic search. To the best of our knowledge, we are the first to extend meta-distillation to a multilingual search application. Our empirical results show that on top of a strong baseline based on sentence transformers, our meta-distillation approach boosts the gains provided by MAML and significantly outperforms naive fine-tuning methods. Furthermore, multilingual meta-distillation learning improves generalization even to unseen languages.