MoLT: Mixture of Layer-Wise Tokens for Efficient Audio-Visual Learning

TL;DR

MoLT introduces a parameter- and memory-efficient framework for audio-visual learning by extracting and fusing layer-wise tokens from late transformer layers, improving performance across multiple benchmarks.

Contribution

The paper presents MoLT, a novel parallel adaptation scheme that replaces heavy sequential adaptation with lightweight, layer-wise token fusion from late transformer layers.

Findings

MoLT outperforms existing methods on audio-visual benchmarks.

Layer-wise token fusion from late layers enhances adaptation performance.

Orthogonality regularization reduces redundancy of latent tokens.

Abstract

In this paper, we propose Mixture of Layer-Wise Tokens (MoLT), a parameter- and memory-efficient adaptation framework for audio-visual learning. The key idea of MoLT is to replace conventional, computationally heavy sequential adaptation at every transformer layer with a parallel, lightweight scheme that extracts and fuses layer-wise tokens only from the late layers. We adopt two types of adapters to distill modality-specific information and cross-modal interaction into compact latent tokens in a layer-wise manner. A token fusion module then dynamically fuses these layer-wise tokens by taking into account their relative significance. To prevent the redundancy of latent tokens, we apply an orthogonality regularization between latent tokens during training. Through the systematic analysis of the position of adaptation in the pre-trained transformers, we extract latent tokens only from the late layers of the transformers. This strategic adaptation approach avoids error propagation from the volatile early-layer features, thereby maximizing the adaptation performance while maintaining parameter and memory efficiency. Through extensive experiments, we demonstrate that MoLT outperforms existing methods on diverse audio-visual benchmarks, including Audio-Visual Question Answering, Audio-Visual Segmentation, and Audio-Visual Event Localization.