Audio Captioning with Composition of Acoustic and Semantic Information

TL;DR

This paper introduces a novel encoder-decoder model for audio captioning that integrates semantic and acoustic information, demonstrating improved performance over existing methods on two datasets.

Contribution

The study presents a new BiGRU-based architecture combining audio and semantic embeddings, with a semantic prediction module for test audio, advancing audio captioning techniques.

Findings

Outperforms state-of-the-art models on Clotho and AudioCaps datasets

Semantic information enhances captioning accuracy

Effective use of multiple audio features

Abstract

Generating audio captions is a new research area that combines audio and natural language processing to create meaningful textual descriptions for audio clips. To address this problem, previous studies mostly use the encoder-decoder based models without considering semantic information. To fill this gap, we present a novel encoder-decoder architecture using bi-directional Gated Recurrent Units (BiGRU) with audio and semantic embeddings. We extract semantic embedding by obtaining subjects and verbs from the audio clip captions and combine these embedding with audio embedding to feed the BiGRU-based encoder-decoder model. To enable semantic embeddings for the test audios, we introduce a Multilayer Perceptron classifier to predict the semantic embeddings of those clips. We also present exhaustive experiments to show the efficiency of different features and datasets for our proposed model the audio captioning task. To extract audio features, we use the log Mel energy features, VGGish embeddings, and a pretrained audio neural network (PANN) embeddings. Extensive experiments on two audio captioning datasets Clotho and AudioCaps show that our proposed model outperforms state-of-the-art audio captioning models across different evaluation metrics and using the semantic information improves the captioning performance. Keywords: Audio captioning; PANNs; VGGish; GRU; BiGRU.