Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigation

TL;DR

This study investigates whether basic linguistic units like phones, syllables, and words can emerge naturally as side-products of audiovisual cross-situational learning in neural networks, without explicit targeting of these units.

Contribution

It introduces the latent language hypothesis (LLH) and evaluates its validity through extensive simulations with neural networks on synthetic and real speech data.

Findings

Latent representations can reflect phonetic, syllabic, or lexical structures.

Audiovisual learning supports emergence of linguistic units.

Neural networks develop meaningful linguistic features without explicit supervision.

Abstract

Decades of research has studied how language learning infants learn to discriminate speech sounds, segment words, and associate words with their meanings. While gradual development of such capabilities is unquestionable, the exact nature of these skills and the underlying mental representations yet remains unclear. In parallel, computational studies have shown that basic comprehension of speech can be achieved by statistical learning between speech and concurrent referentially ambiguous visual input. These models can operate without prior linguistic knowledge such as representations of linguistic units, and without learning mechanisms specifically targeted at such units. This has raised the question of to what extent knowledge of linguistic units, such as phone(me)s, syllables, and words, could actually emerge as latent representations supporting the translation between speech and representations in other modalities, and without the units being proximal learning targets for the learner. In this study, we formulate this idea as the so-called latent language hypothesis (LLH), connecting linguistic representation learning to general predictive processing within and across sensory modalities. We review the extent that the audiovisual aspect of LLH is supported by the existing computational studies. We then explore LLH further in extensive learning simulations with different neural network models for audiovisual cross-situational learning, and comparing learning from both synthetic and real speech data. We investigate whether the latent representations learned by the networks reflect phonetic, syllabic, or lexical structure of input speech by utilizing an array of complementary evaluation metrics related to linguistic selectivity and temporal characteristics of the representations. As a result, we find that representations associated...