With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Models

TL;DR

This paper investigates whether multimodal large language models can understand sound symbolism and iconicity through vision and text, revealing their varying capabilities and dependencies on model size.

Contribution

It provides the first systematic analysis of VLMs and LLMs in recognizing sound symbolism and iconicity, highlighting their potential and limitations.

Findings

VLMs show varying agreement with human judgments

Magnitude symbolism is easier for VLMs than shape symbolism

Model size influences understanding of linguistic iconicity

Abstract

Recently, Large Language Models (LLMs) and Vision Language Models (VLMs) have demonstrated aptitude as potential substitutes for human participants in experiments testing psycholinguistic phenomena. However, an understudied question is to what extent models that only have access to vision and text modalities are able to implicitly understand sound-based phenomena via abstract reasoning from orthography and imagery alone. To investigate this, we analyse the ability of VLMs and LLMs to demonstrate sound symbolism (i.e., to recognise a non-arbitrary link between sounds and concepts) as well as their ability to "hear" via the interplay of the language and vision modules of open and closed-source multimodal models. We perform multiple experiments, including replicating the classic Kiki-Bouba and Mil-Mal shape and magnitude symbolism tasks, and comparing human judgements of linguistic iconicity with that of LLMs. Our results show that VLMs demonstrate varying levels of agreement with human labels, and more task information may be required for VLMs versus their human counterparts for in silico experimentation. We additionally see through higher maximum agreement levels that Magnitude Symbolism is an easier pattern for VLMs to identify than Shape Symbolism, and that an understanding of linguistic iconicity is highly dependent on model size.