[b]=[d]-[t]+[p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic

TL;DR

This paper reveals that self-supervised speech models encode phonological features as linear, compositional vectors, enabling phonological vector arithmetic across 96 languages.

Contribution

It demonstrates that S3Ms encode phonological information in interpretable, linear vectors, and introduces the concept of phonological vector arithmetic in speech representations.

Findings

Linear directions in model space correspond to phonological features.

Scale of phonological vectors correlates with acoustic realization.

Adding and scaling vectors produces phonological continuums.

Abstract

Self-supervised speech models (S3Ms) are known to encode rich phonetic information, yet how this information is structured remains underexplored. We conduct a comprehensive study across 96 languages to analyze the underlying structure of S3M representations, with particular attention to phonological vectors. We first show that there exist linear directions within the model's representation space that correspond to phonological features. We further demonstrate that the scale of these phonological vectors correlate to the degree of acoustic realization of their corresponding phonological features in a continuous manner. For example, the difference between [d] and [t] yields a voicing vector: adding this vector to [p] produces [b], while scaling it results in a continuum of voicing. Together, these findings indicate that S3Ms encode speech using phonologically interpretable and compositional vectors, demonstrating phonological vector arithmetic. All code and interactive demos are available at https://github.com/juice500ml/phonetic-arithmetic .