Leveraging Prediction Entropy for Automatic Prompt Weighting in Zero-Shot Audio-Language Classification

TL;DR

This paper introduces an entropy-based prompt weighting method that enhances zero-shot audio classification by combining prompts to maximize confidence, leading to significant accuracy improvements without additional labels.

Contribution

The paper proposes a novel entropy-guided prompt weighting technique that optimally combines prompts to improve zero-shot audio classification performance.

Findings

Consistent accuracy gains across five audio datasets.

5-fold improvement over classical prompt ensembling methods.

No additional labels or significant computational overhead required.

Abstract

Audio-language models have recently demonstrated strong zero-shot capabilities by leveraging natural-language supervision to classify audio events without labeled training data. Yet, their performance is highly sensitive to the wording of text prompts, with small variations leading to large fluctuations in accuracy. Prior work has mitigated this issue through prompt learning or prompt ensembling. However, these strategies either require annotated data or fail to account for the fact that some prompts may negatively impact performance. In this work, we present an entropy-guided prompt weighting approach that aims to find a robust combination of prompt contributions to maximize prediction confidence. To this end, we formulate a tailored objective function that minimizes prediction entropy to yield new prompt weights, utilizing low-entropy as a proxy for high confidence. Our approach can be applied to individual samples or a batch of audio samples, requiring no additional labels and incurring negligible computational overhead. Experiments on five audio classification datasets covering environmental, urban, and vocal sounds, demonstrate consistent gains compared to classical prompt ensembling methods in a zero-shot setting, with accuracy improvements 5-times larger across the whole benchmark.