Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models

TL;DR

This paper introduces a multi-task learning approach to predict zero-shot performance of multilingual models across languages and tasks, enabling better performance estimation and feature analysis.

Contribution

It presents a novel multi-task learning framework that improves zero-shot performance prediction and identifies key features influencing transfer across tasks.

Findings

Enhanced accuracy in zero-shot performance prediction

Identification of common features affecting transferability

Robust feature selection across multiple tasks

Abstract

Massively Multilingual Transformer based Language Models have been observed to be surprisingly effective on zero-shot transfer across languages, though the performance varies from language to language depending on the pivot language(s) used for fine-tuning. In this work, we build upon some of the existing techniques for predicting the zero-shot performance on a task, by modeling it as a multi-task learning problem. We jointly train predictive models for different tasks which helps us build more accurate predictors for tasks where we have test data in very few languages to measure the actual performance of the model. Our approach also lends us the ability to perform a much more robust feature selection and identify a common set of features that influence zero-shot performance across a variety of tasks.