MoTE: Reconciling Generalization with Specialization for Visual-Language to Video Knowledge Transfer

TL;DR

MoTE introduces a unified framework that balances generalization and specialization in video recognition by tuning multiple temporal experts and employing weight merging regularization, achieving state-of-the-art results.

Contribution

The paper proposes MoTE, a novel model that balances zero-shot and close-set performance through mixture of experts and weight merging regularization.

Findings

Achieves state-of-the-art results on Kinetics-400 & 600 datasets.

Balances zero-shot and close-set video recognition effectively.

Introduces weight merging regularization for expert knowledge preservation.

Abstract

Transferring visual-language knowledge from large-scale foundation models for video recognition has proved to be effective. To bridge the domain gap, additional parametric modules are added to capture the temporal information. However, zero-shot generalization diminishes with the increase in the number of specialized parameters, making existing works a trade-off between zero-shot and close-set performance. In this paper, we present MoTE, a novel framework that enables generalization and specialization to be balanced in one unified model. Our approach tunes a mixture of temporal experts to learn multiple task views with various degrees of data fitting. To maximally preserve the knowledge of each expert, we propose \emph{Weight Merging Regularization}, which regularizes the merging process of experts in weight space. Additionally with temporal feature modulation to regularize the contribution of temporal feature during test. We achieve a sound balance between zero-shot and close-set video recognition tasks and obtain state-of-the-art or competitive results on various datasets, including Kinetics-400 \& 600, UCF, and HMDB. Code is available at \url{https://github.com/ZMHH-H/MoTE}.