Investigating Post-pretraining Representation Alignment for Cross-Lingual Question Answering

TL;DR

This paper explores how post-pretraining representation alignment improves cross-lingual question answering by fine-tuning multilingual models, analyzing data size and language effects, and releasing a new evaluation dataset.

Contribution

It introduces a post-hoc alignment method for multilingual models and provides insights into data and language impacts, along with a new dataset for evaluation.

Findings

Representation alignment improves cross-lingual QA performance.

Data size and language choice significantly affect fine-tuning outcomes.

Public dataset and code facilitate further research in cross-lingual QA.

Abstract

Human knowledge is collectively encoded in the roughly 6500 languages spoken around the world, but it is not distributed equally across languages. Hence, for information-seeking question answering (QA) systems to adequately serve speakers of all languages, they need to operate cross-lingually. In this work we investigate the capabilities of multilingually pre-trained language models on cross-lingual QA. We find that explicitly aligning the representations across languages with a post-hoc fine-tuning step generally leads to improved performance. We additionally investigate the effect of data size as well as the language choice in this fine-tuning step, also releasing a dataset for evaluating cross-lingual QA systems. Code and dataset are publicly available here: https://github.com/ffaisal93/aligned_qa