Do Blind Spots Matter for Word-Referent Mapping? A Computational Study with Infant Egocentric Video

TL;DR

This study introduces a biologically inspired masking strategy for visual representation learning, improving word-referent mapping in infant-like scenarios using egocentric video data.

Contribution

It proposes a novel masking approach based on the human blind spot, enhancing self-supervised visual learning for language acquisition models.

Findings

Biologically plausible masking performs comparably to random masking.

The approach effectively learns word-referent mappings from naturalistic video data.

The method aligns with human visual processing mechanisms.

Abstract

Typically, children start to learn their first words between 6 and 9 months, linking spoken utterances to their visual referents. Without prior knowledge, a word encountered for the first time can be interpreted in countless ways; it might refer to any of the objects in the environment, their components, or attributes. Using longitudinal, egocentric, and ecologically valid data from the experience of one child, in this work, we propose a self-supervised and biologically plausible strategy to learn strong visual representations. Our masked autoencoder-based visual backbone incorporates knowledge about the blind spot in human eyes to define a novel masking strategy. This mask and reconstruct approach attempts to mimic the way the human brain fills the gaps in the eyes' field of view. This represents a significant shift from standard random masking strategies, which are difficult to justify from a biological perspective. The pretrained encoder is utilized in a contrastive learning-based video-text model capable of acquiring word-referent mappings. Extensive evaluation suggests that the proposed biologically plausible masking strategy is at least as effective as random masking for learning word-referent mappings from cross-situational and temporally extended episodes.