Multitasking Models are Robust to Structural Failure: A Neural Model for Bilingual Cognitive Reserve

TL;DR

This paper demonstrates that multitask bilingual models are more robust to neuron failures than monolingual models, supported by empirical experiments and theoretical analysis linking robustness to spectral properties of learned representations.

Contribution

It reveals a novel connection between multitask learning and robustness to neuron perturbations, providing both empirical evidence and a theoretical framework.

Findings

Bilingual models outperform monolingual models under neuron perturbations.

Multitasking creates more robust neural representations.

Theoretical analysis links robustness to spectral properties of representations.

Abstract

We find a surprising connection between multitask learning and robustness to neuron failures. Our experiments show that bilingual language models retain higher performance under various neuron perturbations, such as random deletions, magnitude pruning and weight noise compared to equivalent monolingual ones. We provide a theoretical justification for this robustness by mathematically analyzing linear representation learning and showing that multitasking creates more robust representations. Our analysis connects robustness to spectral properties of the learned representation and proves that multitasking leads to higher robustness for diverse task vectors. We open-source our code and models: https://github.com/giannisdaras/multilingual_robustness