One Instruction Does Not Fit All: How Well Do Embeddings Align Personas and Instructions in Low-Resource Indian Languages?

TL;DR

This paper introduces a comprehensive benchmark for evaluating multilingual embedding models in Indian languages, focusing on persona-instruction alignment across retrieval and classification tasks, to guide model selection and future research.

Contribution

It presents a unified benchmark covering 12 Indian languages with multiple evaluation tasks, and provides baseline results for various multilingual embedding models in this context.

Findings

E5-Large-Instruct achieves highest monolingual retrieval recall@1 (27.4%)

BGE-M3 leads in cross-lingual transfer with 20.7% recall@1

LaBSE attains 75.3% AUROC in classification tasks

Abstract

Aligning multilingual assistants with culturally grounded user preferences is essential for serving India's linguistically diverse population of over one billion speakers across multiple scripts. However, existing benchmarks either focus on a single language or conflate retrieval with generation, leaving open the question of whether current embedding models can encode persona-instruction compatibility without relying on response synthesis. We present a unified benchmark spanning 12 Indian languages and four evaluation tasks: monolingual and cross-lingual persona-to-instruction retrieval, reverse retrieval from instruction to persona, and binary compatibility classification. Eight multilingual embedding models are evaluated in a frozen-encoder setting with a thin logistic regression head for classification. E5-Large-Instruct achieves the highest Recall@1 of 27.4\% on monolingual retrieval and 20.7\% on cross-lingual transfer, while BGE-M3 leads reverse retrieval at 32.1\% Recall@1. For classification, LaBSE attains 75.3\% AUROC with strong calibration. These findings offer practical guidance for model selection in Indic multilingual retrieval and establish reproducible baselines for future work\footnote{Code, datasets, and models are publicly available at https://github.com/aryashah2k/PI-Indic-Align.