Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings

TL;DR

This paper advances visually prompted keyword localisation by introducing a few-shot learning approach for pair mining without transcriptions and demonstrates its application on a low-resource language, Yoruba, with promising but less accurate results.

Contribution

It proposes a novel few-shot learning scheme for pair mining in VPKL without transcriptions and applies it to a low-resource language, Yoruba, expanding the method's applicability.

Findings

Minimal performance drop on English with the new method

Reasonable scores achieved on Yoruba, a low-resource language

Performance decline due to less accurate pair mining in Yoruba

Abstract

Given an image query, visually prompted keyword localisation (VPKL) aims to find occurrences of the depicted word in a speech collection. This can be useful when transcriptions are not available for a low-resource language (e.g. if it is unwritten). Previous work showed that VPKL can be performed with a visually grounded speech model trained on paired images and unlabelled speech. But all experiments were done on English. Moreover, transcriptions were used to get positive and negative pairs for the contrastive loss. This paper introduces a few-shot learning scheme to mine pairs automatically without transcriptions. On English, this results in only a small drop in performance. We also - for the first time - consider VPKL on a real low-resource language, Yoruba. While scores are reasonable, here we see a bigger drop in performance compared to using ground truth pairs because the mining is less accurate in Yoruba.