Leveraging Pre-trained BERT for Audio Captioning

TL;DR

This paper explores using pre-trained BERT as a language decoder in audio captioning systems, combined with PANNs as encoder, demonstrating competitive results and addressing data scarcity issues.

Contribution

It introduces the novel application of pre-trained BERT models as decoders in audio captioning, enhancing performance with minimal additional training.

Findings

BERT-based decoders achieve competitive results on AudioCaps.

Using PANNs as encoders improves audio feature extraction.

Pre-trained BERT models effectively mitigate data scarcity in audio captioning.

Abstract

Audio captioning aims at using natural language to describe the content of an audio clip. Existing audio captioning systems are generally based on an encoder-decoder architecture, in which acoustic information is extracted by an audio encoder and then a language decoder is used to generate the captions. Training an audio captioning system often encounters the problem of data scarcity. Transferring knowledge from pre-trained audio models such as Pre-trained Audio Neural Networks (PANNs) have recently emerged as a useful method to mitigate this issue. However, there is less attention on exploiting pre-trained language models for the decoder, compared with the encoder. BERT is a pre-trained language model that has been extensively used in Natural Language Processing (NLP) tasks. Nevertheless, the potential of BERT as the language decoder for audio captioning has not been investigated. In this study, we demonstrate the efficacy of the pre-trained BERT model for audio captioning. Specifically, we apply PANNs as the encoder and initialize the decoder from the public pre-trained BERT models. We conduct an empirical study on the use of these BERT models for the decoder in the audio captioning model. Our models achieve competitive results with the existing audio captioning methods on the AudioCaps dataset.