# Leveraging Context for Perceptual Prediction Using Word Embeddings

## TL;DR

This paper explores whether word embeddings, with or without context, can predict perceptual qualities like brightness and shape from language data.

## Contribution

The study introduces a novel comparison of decontextualized and contextualized embeddings for predicting perceptual features.

## Key findings

- Shape predictions using embeddings were generally accurate, while brightness predictions were more modest.
- Adding context did not significantly improve perceptual prediction performance.
- Adjective-noun phrases showed good perceptual prediction, reflecting nonadditive brightness in embeddings.

## Abstract

Word embeddings derived from large language corpora have been successfully used in cognitive science and artificial intelligence to represent linguistic meaning. However, there is continued debate as to how well they encode useful information about the perceptual qualities of concepts. This debate is critical to identifying the scope of embodiment in human semantics. If perceptual object properties can be inferred from word embeddings derived from language alone, this suggests that language provides a useful adjunct to direct perceptual experience for acquiring this kind of conceptual knowledge. Previous research has shown mixed performance when embeddings are used to predict perceptual qualities. Here, we tested if we could improve performance by leveraging the ability of Transformer‐based language models to represent word meaning in context. To this end, we conducted two experiments. Our first experiment investigated noun representations. We generated decontextualized (“charcoal”) and contextualized (“the brightness of charcoal”) Word2Vec and BERT embeddings for a large set of concepts and compared their ability to predict human ratings of the concepts’ brightness. We repeated this procedure to also probe for the shape of those concepts. In general, we found very good prediction performance for shape, and a more modest performance for brightness. The addition of context did not improve perceptual prediction performance. In Experiment 2, we investigated representations of adjective–noun phrases. Perceptual prediction performance was generally found to be good, with the nonadditive nature of adjective brightness reflected in the word embeddings. We also found that the addition of context had a limited impact on how well perceptual features could be predicted. We frame these results against current work on the interpretability of language models and debates surrounding embodiment in human conceptual processing.