Object Referring in Visual Scene with Spoken Language

TL;DR

This paper introduces a new approach for object referring in visual scenes using spoken language, supported by two datasets and methods that outperform existing techniques, even under noisy audio conditions.

Contribution

It presents the first datasets and a novel multi-level vision-language interaction approach for ORSpoken, addressing the gap in spoken language-based object referring.

Findings

Our method outperforms existing approaches significantly.

The approach remains effective under audio noise conditions.

Datasets facilitate multi-modality learning for ORSpoken.

Abstract

Object referring has important applications, especially for human-machine interaction. While having received great attention, the task is mainly attacked with written language (text) as input rather than spoken language (speech), which is more natural. This paper investigates Object Referring with Spoken Language (ORSpoken) by presenting two datasets and one novel approach. Objects are annotated with their locations in images, text descriptions and speech descriptions. This makes the datasets ideal for multi-modality learning. The approach is developed by carefully taking down ORSpoken problem into three sub-problems and introducing task-specific vision-language interactions at the corresponding levels. Experiments show that our method outperforms competing methods consistently and significantly. The approach is also evaluated in the presence of audio noise, showing the efficacy of the proposed vision-language interaction methods in counteracting background noise.