Beyond Training: Dynamic Token Merging for Zero-Shot Video Understanding

TL;DR

DYTO introduces a dynamic token merging framework that adaptively compresses video tokens, enhancing zero-shot video understanding performance without fine-tuning, by balancing efficiency and semantic preservation.

Contribution

It presents a novel, training-free approach with hierarchical frame selection and bipartite token merging to improve zero-shot video understanding.

Findings

Outperforms existing methods on multiple benchmarks.

Sets new state-of-the-art for zero-shot video understanding.

Efficiently balances computational cost with semantic detail.

Abstract

Recent advancements in multimodal large language models (MLLMs) have opened new avenues for video understanding. However, achieving high fidelity in zero-shot video tasks remains challenging. Traditional video processing methods rely heavily on fine-tuning to capture nuanced spatial-temporal details, which incurs significant data and computation costs. In contrast, training-free approaches, though efficient, often lack robustness in preserving context-rich features across complex video content. To this end, we propose DYTO, a novel dynamic token merging framework for zero-shot video understanding that adaptively optimizes token efficiency while preserving crucial scene details. DYTO integrates a hierarchical frame selection and a bipartite token merging strategy to dynamically cluster key frames and selectively compress token sequences, striking a balance between computational efficiency with semantic richness. Extensive experiments across multiple benchmarks demonstrate the effectiveness of DYTO, achieving superior performance compared to both fine-tuned and training-free methods and setting a new state-of-the-art for zero-shot video understanding.