Textual Supervision for Visually Grounded Spoken Language Understanding

TL;DR

This paper compares end-to-end and pipeline-based visually grounded spoken language understanding models, highlighting the effectiveness of pipeline approaches with ample text and exploring translations as alternatives to transcriptions in low-resource scenarios.

Contribution

It provides a comparative analysis of end-to-end versus pipeline approaches and investigates the use of translations as substitutes for transcriptions in low-resource settings.

Findings

Pipeline approach outperforms end-to-end with sufficient text data.

Translations can substitute transcriptions but require more data.

End-to-end models benefit from transcriptions during training.

Abstract

Visually-grounded models of spoken language understanding extract semantic information directly from speech, without relying on transcriptions. This is useful for low-resource languages, where transcriptions can be expensive or impossible to obtain. Recent work showed that these models can be improved if transcriptions are available at training time. However, it is not clear how an end-to-end approach compares to a traditional pipeline-based approach when one has access to transcriptions. Comparing different strategies, we find that the pipeline approach works better when enough text is available. With low-resource languages in mind, we also show that translations can be effectively used in place of transcriptions but more data is needed to obtain similar results.