Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation

TL;DR

Mettle introduces a memory-efficient, meta-token learning approach for adapting large-scale transformer models to audio-visual tasks, enabling effective task-specific adaptation with reduced resource requirements.

Contribution

The paper proposes a novel Layer-Centric Distillation and Meta-Token Injection framework for efficient audio-visual model adaptation, improving memory usage and training speed.

Findings

Reduces memory consumption and training time significantly.

Maintains competitive accuracy on multiple benchmarks.

Supports both classification and segmentation tasks.

Abstract

We present \textbf{Met}a-\textbf{T}oken \textbf{Le}arning (Mettle), a simple and memory-efficient method for adapting large-scale pretrained transformer models to downstream audio-visual tasks. Instead of sequentially modifying the output feature distribution of the transformer backbone, Mettle utilizes a lightweight \textit{Layer-Centric Distillation (LCD)} module to distill in parallel the intact audio or visual features embedded by each transformer layer into compact meta-tokens. This distillation process considers both pretrained knowledge preservation and task-specific adaptation. The obtained meta-tokens can be directly applied to classification tasks, such as audio-visual event localization and audio-visual video parsing. To further support fine-grained segmentation tasks, such as audio-visual segmentation, we introduce a \textit{Meta-Token Injection (MTI)} module, which utilizes the audio and visual meta-tokens distilled from the top transformer layer to guide feature adaptation in earlier layers. Extensive experiments on multiple audiovisual benchmarks demonstrate that our method significantly reduces memory usage and training time while maintaining parameter efficiency and competitive accuracy.