Audio Difference Captioning Utilizing Similarity-Discrepancy Disentanglement

TL;DR

This paper introduces Audio Difference Captioning (ADC), a novel task that describes differences between similar audio clips, and proposes a transformer-based model with disentanglement to improve difference extraction, validated on a new dataset.

Contribution

It presents the ADC task, a new dataset, and a transformer model with similarity-discrepancy disentanglement for effective difference captioning in audio.

Findings

The proposed model effectively describes audio differences.

Attention visualization confirms improved focus on differences.

The AudioDiffCaps dataset supports ADC research.

Abstract

We proposed Audio Difference Captioning (ADC) as a new extension task of audio captioning for describing the semantic differences between input pairs of similar but slightly different audio clips. The ADC solves the problem that conventional audio captioning sometimes generates similar captions for similar audio clips, failing to describe the difference in content. We also propose a cross-attention-concentrated transformer encoder to extract differences by comparing a pair of audio clips and a similarity-discrepancy disentanglement to emphasize the difference in the latent space. To evaluate the proposed methods, we built an AudioDiffCaps dataset consisting of pairs of similar but slightly different audio clips with human-annotated descriptions of their differences. The experiment with the AudioDiffCaps dataset showed that the proposed methods solve the ADC task effectively and improve the attention weights to extract the difference by visualizing them in the transformer encoder.