How Contrastive Decoding Enhances Large Audio Language Models?

TL;DR

This paper systematically evaluates contrastive decoding strategies for large audio language models, introducing a framework to understand their effectiveness and guiding model selection based on error profiles.

Contribution

It identifies the most effective contrastive decoding strategies, introduces a Transition Matrix framework to analyze error correction, and provides guidelines for model suitability.

Findings

Audio-Aware Decoding and Audio Contrastive Decoding are most effective.

Contrastive Decoding corrects errors related to audio absence and uncertainty.

It does not improve flawed reasoning or confident errors.

Abstract

While Contrastive Decoding (CD) has proven effective at enhancing Large Audio Language Models (LALMs), the underlying mechanisms driving its success and the comparative efficacy of different strategies remain unclear. This study systematically evaluates four distinct CD strategies across diverse LALM architectures. We identify Audio-Aware Decoding and Audio Contrastive Decoding as the most effective methods. However, their impact varies significantly by model. To explain this variability, we introduce a Transition Matrix framework to map error pattern shifts during inference. Our analysis demonstrates that CD reliably rectifies errors in which models falsely claim an absence of audio or resort to uncertainty-driven guessing. Conversely, it fails to correct flawed reasoning or confident misassertions. Ultimately, these findings provide a clear guideline for determining which LALM architectures are most suitable for CD enhancement based on their baseline error profiles.