The mutual exclusivity bias of bilingual visually grounded speech models

TL;DR

This paper investigates the mutual exclusivity bias in bilingual visually grounded speech models, revealing that bilingual models generally show a weaker bias than monolingual ones, with implications for understanding language learning and cross-lingual ambiguity.

Contribution

It introduces bilingual VGS models trained on multiple languages, analyzes their ME bias, and provides insights into the factors influencing this bias in multilingual settings.

Findings

Bilingual models exhibit a weaker ME bias than monolingual models.

Combined visual embeddings in bilingual models have less variance for familiar data.

The study offers new explanations for the origin of ME bias in VGS models.

Abstract

Mutual exclusivity (ME) is a strategy where a novel word is associated with a novel object rather than a familiar one, facilitating language learning in children. Recent work has found an ME bias in a visually grounded speech (VGS) model trained on English speech with paired images. But ME has also been studied in bilingual children, who may employ it less due to cross-lingual ambiguity. We explore this pattern computationally using bilingual VGS models trained on combinations of English, French, and Dutch. We find that bilingual models generally exhibit a weaker ME bias than monolingual models, though exceptions exist. Analyses show that the combined visual embeddings of bilingual models have a smaller variance for familiar data, partly explaining the increase in confusion between novel and familiar concepts. We also provide new insights into why the ME bias exists in VGS models in the first place. Code and data: https://github.com/danoneata/me-vgs