Meta-Learning for Effective Multi-task and Multilingual Modelling

TL;DR

This paper introduces a meta-learning approach to enhance multi-task and multilingual NLP models by learning interactions across tasks and languages, improving performance and enabling effective zero-shot transfer.

Contribution

The paper proposes a novel meta-learning framework that models interactions between tasks and languages, and explores sampling strategies to improve multilingual NLP performance.

Findings

Meta-learning improves multi-task multilingual NLP performance.

The approach outperforms baseline models on the XTREME benchmark.

Zero-shot transfer to unseen languages is effectively demonstrated.

Abstract

Natural language processing (NLP) tasks (e.g. question-answering in English) benefit from knowledge of other tasks (e.g. named entity recognition in English) and knowledge of other languages (e.g. question-answering in Spanish). Such shared representations are typically learned in isolation, either across tasks or across languages. In this work, we propose a meta-learning approach to learn the interactions between both tasks and languages. We also investigate the role of different sampling strategies used during meta-learning. We present experiments on five different tasks and six different languages from the XTREME multilingual benchmark dataset. Our meta-learned model clearly improves in performance compared to competitive baseline models that also include multi-task baselines. We also present zero-shot evaluations on unseen target languages to demonstrate the utility of our proposed model.