A Whisper transformer for audio captioning trained with synthetic captions and transfer learning

TL;DR

This paper introduces an audio captioning approach using a pretrained Whisper transformer model trained with synthetic captions and transfer learning, showing how various training strategies affect performance.

Contribution

It presents a novel application of the Whisper transformer for audio captioning, leveraging synthetic data and transfer learning to improve results.

Findings

Different training strategies significantly impact model performance

Pretraining on synthetic captions enhances captioning accuracy

Model size and dataset mixture influence results

Abstract

The field of audio captioning has seen significant advancements in recent years, driven by the availability of large-scale audio datasets and advancements in deep learning techniques. In this technical report, we present our approach to audio captioning, focusing on the use of a pretrained speech-to-text Whisper model and pretraining on synthetic captions. We discuss our training procedures and present our experiments' results, which include model size variations, dataset mixtures, and other hyperparameters. Our findings demonstrate the impact of different training strategies on the performance of the audio captioning model. Our code and trained models are publicly available on GitHub and Hugging Face Hub.