Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech

TL;DR

This paper demonstrates that grounded language understanding can be achieved directly from raw speech and visual percepts, reducing reliance on textual data and enhancing inclusivity in language grounding systems.

Contribution

It introduces a method for grounded language learning from paired visual and raw speech inputs, leveraging self-supervised speech models to improve inclusivity and performance.

Findings

Speech representations enable effective grounded language learning.

The approach reduces demographic bias in language grounding.

Performance is maintained or improved with raw speech inputs.

Abstract

Learning to understand grounded language, which connects natural language to percepts, is a critical research area. Prior work in grounded language acquisition has focused primarily on textual inputs. In this work we demonstrate the feasibility of performing grounded language acquisition on paired visual percepts and raw speech inputs. This will allow interactions in which language about novel tasks and environments is learned from end users, reducing dependence on textual inputs and potentially mitigating the effects of demographic bias found in widely available speech recognition systems. We leverage recent work in self-supervised speech representation models and show that learned representations of speech can make language grounding systems more inclusive towards specific groups while maintaining or even increasing general performance.