Tonguescape: Exploring Language Models Understanding of Vowel Articulation

TL;DR

This paper investigates whether vision-based language models can understand vowel articulation by associating tongue positions with speech sounds, using datasets derived from MRI images and videos.

Contribution

It introduces a new approach to evaluate vision-language models' understanding of vowel articulation through visual datasets and analysis.

Findings

Models understand vowels better with reference examples

Models struggle to infer tongue positions without visual references

Potential for multimodal models to grasp speech articulation mechanisms

Abstract

Vowels are primarily characterized by tongue position. Humans have discovered these features of vowel articulation through their own experience and explicit objective observation such as using MRI. With this knowledge and our experience, we can explain and understand the relationship between tongue positions and vowels, and this knowledge is helpful for language learners to learn pronunciation. Since language models (LMs) are trained on a large amount of data that includes linguistic and medical fields, our preliminary studies indicate that an LM is able to explain the pronunciation mechanisms of vowels. However, it is unclear whether multi-modal LMs, such as vision LMs, align textual information with visual information. One question arises: do LMs associate real tongue positions with vowel articulation? In this study, we created video and image datasets from the existing real-time MRI dataset and investigated whether LMs can understand vowel articulation based on tongue positions using vision-based information. Our findings suggest that LMs exhibit potential for understanding vowels and tongue positions when reference examples are provided while they have difficulties without them. Our code for dataset building is available on GitHub.