Modelling word learning and recognition using visually grounded speech

TL;DR

This paper explores a visually grounded speech model that learns to recognize words from visual and spoken input without prior knowledge, demonstrating human-like word recognition effects and analyzing the impact of vector quantisation.

Contribution

It investigates the time-course of word recognition in a grounded speech model and assesses the role of vector quantisation in word discovery and recognition.

Findings

Model recognizes nouns and differentiates singular/plural forms.

Recognition influenced by word competition effects similar to humans.

Vector quantisation does not improve word recognition and may require more input.

Abstract

Background: Computational models of speech recognition often assume that the set of target words is already given. This implies that these models do not learn to recognise speech from scratch without prior knowledge and explicit supervision. Visually grounded speech models learn to recognise speech without prior knowledge by exploiting statistical dependencies between spoken and visual input. While it has previously been shown that visually grounded speech models learn to recognise the presence of words in the input, we explicitly investigate such a model as a model of human speech recognition. Methods: We investigate the time-course of word recognition as simulated by the model using a gating paradigm to test whether its recognition is affected by well-known word-competition effects in human speech processing. We furthermore investigate whether vector quantisation, a technique for discrete representation learning, aids the model in the discovery and recognition of words. Results/Conclusion: Our experiments show that the model is able to recognise nouns in isolation and even learns to properly differentiate between plural and singular nouns. We also find that recognition is influenced by word competition from the word-initial cohort and neighbourhood density, mirroring word competition effects in human speech comprehension. Lastly, we find no evidence that vector quantisation is helpful in discovering and recognising words. Our gating experiments even show that the vector quantised model requires more of the input sequence for correct recognition.