Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding

TL;DR

This paper improves automated audio captioning by integrating optimized audio encoding, large language models, and error correction, achieving state-of-the-art results in descriptive accuracy and robustness.

Contribution

It introduces a novel AAC framework combining a pre-trained audio encoder with ensemble distillation, a Llama 2 decoder, and a text correction LLM, all optimized with LoRA.

Findings

Achieved 33.0 SPIDEr-FL score, surpassing previous bests.

Demonstrated effectiveness of ensemble distillation and Llama 2 in AAC.

Validated improvements through extensive experiments.

Abstract

Automated audio captioning (AAC) is an audio-to-text task to describe audio contents in natural language. Recently, the advancements in large language models (LLMs), with improvements in training approaches for audio encoders, have opened up possibilities for improving AAC. Thus, we explore enhancing AAC from three aspects: 1) a pre-trained audio encoder via consistent ensemble distillation (CED) is used to improve the effectivity of acoustic tokens, with a querying transformer (Q-Former) bridging the modality gap to LLM and compress acoustic tokens; 2) we investigate the advantages of using a Llama 2 with 7B parameters as the decoder; 3) another pre-trained LLM corrects text errors caused by insufficient training data and annotation ambiguities. Both the audio encoder and text decoder are optimized by low-rank adaptation (LoRA). Experiments show that each of these enhancements is effective. Our method obtains a 33.0 SPIDEr-FL score, outperforming the winner of DCASE 2023 Task 6A.