ASK: Adaptive Self-improving Knowledge Framework for Audio Text Retrieval

TL;DR

The paper introduces ASK, a framework that enhances audio-text retrieval by overcoming local optimization limitations and aligning external knowledge with evolving models, leading to state-of-the-art results.

Contribution

ASK innovatively combines multi-grained knowledge injection and dynamic refinement to address Gradient Locality Bottleneck and Representation-Drift Mismatch in ATR.

Findings

Achieves new state-of-the-art performance on multiple benchmarks.

Effectively mitigates acoustic ambiguities and long-tail concept learning.

Demonstrates robustness across various backbone architectures.

Abstract

The dominant paradigm for Audio-Text Retrieval (ATR) relies on dual-encoder architectures optimized via mini-batch contrastive learning. However, restricting optimization to local in-batch samples creates a fundamental limitation we term the Gradient Locality Bottleneck (GLB), which prevents the resolution of acoustic ambiguities and hinders the learning of rare long-tail concepts. While external knowledge injection can break this bottleneck, it often triggers a problem called Representation-Drift Mismatch (RDM), where a static knowledge base becomes misaligned with evolving encoders, degrading guidance into noise. To address these intertwined challenges, we propose the Adaptive Self-improving Knowledge (ASK) framework. ASK breaks the GLB via multi-grained knowledge injection and mitigates RDM through a dynamic refinement strategy that synchronizes the knowledge base with the model. Additionally, an adaptive reliability weighting scheme is employed to filter retrieval noise based on cross-modal consistency. Extensive experiments across multiple benchmarks demonstrate that ASK consistently achieves new state-of-the-art performance across various backbones.