Automated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularization

TL;DR

This paper introduces a novel approach for automated audio captioning that combines transfer learning with a new self-supervised regularization technique, significantly improving performance on standard benchmarks.

Contribution

It proposes a new architecture leveraging pretrained audio neural networks and introduces RLSSR, a self-supervised regularization method, to enhance captioning accuracy.

Findings

Surpassed state-of-the-art results on Clotho dataset

Effective use of transfer learning for audio captioning

RLSSR improves model embedding quality

Abstract

In this paper, we examine the use of Transfer Learning using Pretrained Audio Neural Networks (PANNs), and propose an architecture that is able to better leverage the acoustic features provided by PANNs for the Automated Audio Captioning Task. We also introduce a novel self-supervised objective, Reconstruction Latent Space Similarity Regularization (RLSSR). The RLSSR module supplements the training of the model by minimizing the similarity between the encoder and decoder embedding. The combination of both methods allows us to surpass state of the art results by a significant margin on the Clotho dataset across several metrics and benchmarks.