Attention-Based Keyword Localisation in Speech using Visual Grounding

TL;DR

This paper explores whether attention mechanisms in visually grounded speech models can improve keyword localisation within speech utterances without explicit supervision, showing significant performance gains despite some semantic confusions.

Contribution

It demonstrates that incorporating attention into convolutional models enhances keyword localisation in visually grounded speech without explicit alignment supervision.

Findings

Attention improves localisation performance significantly.

Semantic confusions are a common source of errors.

Absolute localisation performance remains modest.

Abstract

Visually grounded speech models learn from images paired with spoken captions. By tagging images with soft text labels using a trained visual classifier with a fixed vocabulary, previous work has shown that it is possible to train a model that can detect whether a particular text keyword occurs in speech utterances or not. Here we investigate whether visually grounded speech models can also do keyword localisation: predicting where, within an utterance, a given textual keyword occurs without any explicit text-based or alignment supervision. We specifically consider whether incorporating attention into a convolutional model is beneficial for localisation. Although absolute localisation performance with visually supervised models is still modest (compared to using unordered bag-of-word text labels for supervision), we show that attention provides a large gain in performance over previous visually grounded models. As in many other speech-image studies, we find that many of the incorrect localisations are due to semantic confusions, e.g. locating the word 'backstroke' for the query keyword 'swimming'.