Dealing with Semantic Underspecification in Multimodal NLP

TL;DR

This paper discusses the challenge of semantic underspecification in multimodal NLP, emphasizing its importance for human-like language understanding and highlighting current models' struggles with this phenomenon.

Contribution

It highlights the significance of semantic underspecification in language understanding and advocates for research directions to improve multimodal systems' handling of this feature.

Findings

Multimodal systems struggle with semantic underspecification.

Semantic underspecification is crucial for efficient human communication.

Addressing this can improve NLP system performance and safety.

Abstract

Intelligent systems that aim at mastering language as humans do must deal with its semantic underspecification, namely, the possibility for a linguistic signal to convey only part of the information needed for communication to succeed. Consider the usages of the pronoun they, which can leave the gender and number of its referent(s) underspecified. Semantic underspecification is not a bug but a crucial language feature that boosts its storage and processing efficiency. Indeed, human speakers can quickly and effortlessly integrate semantically-underspecified linguistic signals with a wide range of non-linguistic information, e.g., the multimodal context, social or cultural conventions, and shared knowledge. Standard NLP models have, in principle, no or limited access to such extra information, while multimodal systems grounding language into other modalities, such as vision, are naturally equipped to account for this phenomenon. However, we show that they struggle with it, which could negatively affect their performance and lead to harmful consequences when used for applications. In this position paper, we argue that our community should be aware of semantic underspecification if it aims to develop language technology that can successfully interact with human users. We discuss some applications where mastering it is crucial and outline a few directions toward achieving this goal.