Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioning

TL;DR

This paper introduces an unbiased sliced Wasserstein RBF kernel with rotary positional embedding for audio captioning, effectively capturing temporal relationships and improving caption quality, diversity, and reasoning in audio-language tasks.

Contribution

The paper proposes a novel USW-RBF kernel with rotary embeddings that preserves temporal information and enables efficient optimization for high-quality audio captioning and reasoning.

Findings

Significant improvement in caption quality and diversity on AudioCaps and Clotho datasets.

Enhanced reasoning capabilities in large audio language models with the new kernel.

4% increase in reasoning accuracy on MMAU-test-mini benchmarks.

Abstract

Audio captioning systems face a fundamental challenge: teacher-forcing training creates exposure bias that leads to caption degeneration during inference. While contrastive methods have been proposed as solutions, they typically fail to capture the crucial temporal relationships between acoustic and linguistic modalities. We address this limitation by introducing the unbiased sliced Wasserstein RBF (USW-RBF) kernel with rotary positional embedding, specifically designed to preserve temporal information across modalities. Our approach offers a practical advantage: the kernel enables efficient stochastic gradient optimization, making it computationally feasible for real-world applications. Building on this foundation, we develop a complete audio captioning framework that integrates stochastic decoding to further mitigate caption degeneration. Extensive experiments on AudioCaps and Clotho datasets demonstrate that our method significantly improves caption quality, lexical diversity, and text-to-audio retrieval accuracy. Furthermore, we demonstrate the generalizability of our USW-RBF kernel by applying it to audio reasoning tasks, where it enhances the reasoning capabilities of large audio language models on the CompA-R in terms of correctness and quality. Our kernel also improves the reasoning accuracy of the MMAU-test-mini benchmarks by $4\%$. These results establish our approach as a powerful and generalizable solution for cross-modal alignment challenges in audio-language tasks.