# Sound-Word2Vec: Learning Word Representations Grounded in Sounds

## TL;DR

This paper introduces sound-word2vec, a novel embedding method that grounds word representations in sounds, enhancing aural reasoning in textual tasks and outperforming previous models on sound-related datasets.

## Contribution

The work presents a new sound-grounded embedding scheme for words, enabling better understanding of sound-related concepts and improving performance on aural reasoning tasks.

## Key findings

- Embeddings capture sound similarities between concepts like leaves and paper.
- Outperforms prior models on AMEN and ASLex datasets.
- Useful for sound retrieval and Foley sound discovery in multimedia applications.

## Abstract

To be able to interact better with humans, it is crucial for machines to understand sound - a primary modality of human perception. Previous works have used sound to learn embeddings for improved generic textual similarity assessment. In this work, we treat sound as a first-class citizen, studying downstream textual tasks which require aural grounding. To this end, we propose sound-word2vec - a new embedding scheme that learns specialized word embeddings grounded in sounds. For example, we learn that two seemingly (semantically) unrelated concepts, like leaves and paper are similar due to the similar rustling sounds they make. Our embeddings prove useful in textual tasks requiring aural reasoning like text-based sound retrieval and discovering foley sound effects (used in movies). Moreover, our embedding space captures interesting dependencies between words and onomatopoeia and outperforms prior work on aurally-relevant word relatedness datasets such as AMEN and ASLex.