EmotionCaps: Enhancing Audio Captioning Through Emotion-Augmented Data Generation

TL;DR

EmotionCaps introduces an emotion-augmented dataset for audio captioning, leveraging emotional context to generate higher-quality descriptions and improve model performance in environmental sound understanding.

Contribution

The paper presents a novel dataset with emotion-augmented synthetic captions and demonstrates how emotional information enhances audio captioning models.

Findings

Emotion-augmented data improves caption quality.

Models trained with EmotionCaps outperform baselines.

Emotional context aligns captions better with audio content.

Abstract

Recent progress in audio-language modeling, such as automated audio captioning, has benefited from training on synthetic data generated with the aid of large-language models. However, such approaches for environmental sound captioning have primarily focused on audio event tags and have not explored leveraging emotional information that may be present in recordings. In this work, we explore the benefit of generating emotion-augmented synthetic audio caption data by instructing ChatGPT with additional acoustic information in the form of estimated soundscape emotion. To do so, we introduce EmotionCaps, an audio captioning dataset comprised of approximately 120,000 audio clips with paired synthetic descriptions enriched with soundscape emotion recognition (SER) information. We hypothesize that this additional information will result in higher-quality captions that match the emotional tone of the audio recording, which will, in turn, improve the performance of captioning models trained with this data. We test this hypothesis through both objective and subjective evaluation, comparing models trained with the EmotionCaps dataset to multiple baseline models. Our findings challenge current approaches to captioning and suggest new directions for developing and assessing captioning models.