Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction

TL;DR

This paper introduces an ensemble approach combining language models and image data, leveraging concreteness scores to improve the prediction of perceptual noun properties, outperforming text-only models.

Contribution

It proposes a novel ensemble model that integrates image-derived perceptual properties with language model knowledge, calibrated by concreteness scores.

Findings

Ensemble model significantly improves property prediction accuracy.

Using images enhances the extraction of perceptual noun properties.

Concreteness scores effectively calibrate contributions from text and images.

Abstract

Neural language models encode rich knowledge about entities and their relationships which can be extracted from their representations using probing. Common properties of nouns (e.g., red strawberries, small ant) are, however, more challenging to extract compared to other types of knowledge because they are rarely explicitly stated in texts. We hypothesize this to mainly be the case for perceptual properties which are obvious to the participants in the communication. We propose to extract these properties from images and use them in an ensemble model, in order to complement the information that is extracted from language models. We consider perceptual properties to be more concrete than abstract properties (e.g., interesting, flawless). We propose to use the adjectives' concreteness score as a lever to calibrate the contribution of each source (text vs. images). We evaluate our ensemble model in a ranking task where the actual properties of a noun need to be ranked higher than other non-relevant properties. Our results show that the proposed combination of text and images greatly improves noun property prediction compared to powerful text-based language models.