Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolism

TL;DR

This study explores how multimodal large language models interpret sound-meaning associations, revealing their phonetic intuitions and attention patterns aligned with linguistic research across multiple languages and semantic dimensions.

Contribution

It introduces LEX-ICON, a large multilingual dataset, and provides the first large-scale analysis of phonetic iconicity in MLLMs across text and audio modalities.

Findings

MLLMs show phonetic intuitions consistent with linguistic research

Models' attention patterns focus on iconic phonemes

Results bridge AI and cognitive linguistics

Abstract

Sound symbolism is a linguistic concept that refers to non-arbitrary associations between phonetic forms and their meanings. We suggest that this can be a compelling probe into how Multimodal Large Language Models (MLLMs) interpret auditory information in human languages. We investigate MLLMs' performance on phonetic iconicity across textual (orthographic and IPA) and auditory forms of inputs with up to 25 semantic dimensions (e.g., sharp vs. round), observing models' layer-wise information processing by measuring phoneme-level attention fraction scores. To this end, we present LEX-ICON, an extensive mimetic word dataset consisting of 8,052 words from four natural languages (English, French, Japanese, and Korean) and 2,930 systematically constructed pseudo-words, annotated with semantic features applied across both text and audio modalities. Our key findings demonstrate (1) MLLMs' phonetic intuitions that align with existing linguistic research across multiple semantic dimensions and (2) phonosemantic attention patterns that highlight models' focus on iconic phonemes. These results bridge domains of artificial intelligence and cognitive linguistics, providing the first large-scale, quantitative analyses of phonetic iconicity in terms of MLLMs' interpretability.