Multiple Consistency-guided Test-Time Adaptation for Contrastive Audio-Language Models with Unlabeled Audio

TL;DR

This paper introduces a novel test-time adaptation method for contrastive audio-language models that leverages multiple consistency-guided prompt learning techniques to improve zero-shot domain performance without requiring labeled data.

Contribution

It proposes a new multiple consistency-guided prompt learning framework for TTA in ALMs, enhancing zero-shot performance without annotations.

Findings

Achieves an average of 4.41% performance improvement over state-of-the-art.

Effective across 12 diverse downstream tasks.

Utilizes multiple guidance strategies for robust adaptation.

Abstract

One fascinating aspect of pre-trained Audio-Language Models (ALMs) learning is their impressive zero-shot generalization capability and test-time adaptation (TTA) methods aiming to improve domain performance without annotations. However, previous test time adaptation (TTA) methods for ALMs in zero-shot classification tend to be stuck in incorrect model predictions. In order to further boost the performance, we propose multiple guidance on prompt learning without annotated labels. First, guidance of consistency on both context tokens and domain tokens of ALMs is set. Second, guidance of both consistency across multiple augmented views of each single test sample and contrastive learning across different test samples is set. Third, we propose a corresponding end-end learning framework for the proposed test-time adaptation method without annotated labels. We extensively evaluate our approach on 12 downstream tasks across domains, our proposed adaptation method leads to 4.41% (max 7.50%) average zero-shot performance improvement in comparison with the state-of-the-art models.