EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance

TL;DR

This paper analyzes and enhances the EnCLAP framework for automated audio captioning by modifying components, experimenting with pretraining datasets, and implementing reranking, resulting in a significantly improved model called EnCLAP++.

Contribution

It introduces EnCLAP++, an improved audio captioning model, through systematic analysis and optimization of the original EnCLAP framework.

Findings

EnCLAP++ outperforms the original EnCLAP in caption quality.

Modifications to the acoustic encoder improve captioning performance.

Pretraining with larger datasets enhances model effectiveness.

Abstract

In this work, we aim to analyze and optimize the EnCLAP framework, a state-of-the-art model in automated audio captioning. We investigate the impact of modifying the acoustic encoder components, explore pretraining with different dataset scales, and study the effectiveness of a reranking scheme. Through extensive experimentation and quantitative analysis of generated captions, we develop EnCLAP++, an enhanced version that significantly surpasses the original.