Language in a (Search) Box: Grounding Language Learning in Real-World Human-Machine Interaction

TL;DR

This paper presents a method for grounded language learning from real-world user interactions with search engines, enabling semantic generalization and compositionality without explicit labels, outperforming non-grounded models.

Contribution

It introduces a novel approach to learn grounded semantics from natural human-machine interactions, demonstrating improved compositionality and zero-shot inference capabilities.

Findings

Grounded semantics exhibit compositional properties.

Our model outperforms SOTA non-grounded models on benchmarks.

The approach enables learning without explicit labeling.

Abstract

We investigate grounded language learning through real-world data, by modelling a teacher-learner dynamics through the natural interactions occurring between users and search engines; in particular, we explore the emergence of semantic generalization from unsupervised dense representations outside of synthetic environments. A grounding domain, a denotation function and a composition function are learned from user data only. We show how the resulting semantics for noun phrases exhibits compositional properties while being fully learnable without any explicit labelling. We benchmark our grounded semantics on compositionality and zero-shot inference tasks, and we show that it provides better results and better generalizations than SOTA non-grounded models, such as word2vec and BERT.