DynFocus: Dynamic Cooperative Network Empowers LLMs with Video Understanding

TL;DR

DynFocus introduces a dynamic encoding framework for LLM-based video understanding, effectively balancing detailed information preservation with memory efficiency by selectively encoding frames based on relevance.

Contribution

The paper proposes DynFocus, a novel dynamic cooperative network with modules for adaptive frame selection and encoding, improving memory efficiency in video question answering.

Findings

Achieves competitive performance on five benchmarks.

Effectively reduces token usage while maintaining accuracy.

Demonstrates the benefit of dynamic encoding in video understanding.

Abstract

The challenge in LLM-based video understanding lies in preserving visual and semantic information in long videos while maintaining a memory-affordable token count. However, redundancy and correspondence in videos have hindered the performance potential of existing methods. Through statistical learning on current datasets, we observe that redundancy occurs in both repeated and answer-irrelevant frames, and the corresponding frames vary with different questions. This suggests the possibility of adopting dynamic encoding to balance detailed video information preservation with token budget reduction. To this end, we propose a dynamic cooperative network, DynFocus, for memory-efficient video encoding in this paper. Specifically, i) a Dynamic Event Prototype Estimation (DPE) module to dynamically select meaningful frames for question answering; (ii) a Compact Cooperative Encoding (CCE) module that encodes meaningful frames with detailed visual appearance and the remaining frames with sketchy perception separately. We evaluate our method on five publicly available benchmarks, and experimental results consistently demonstrate that our method achieves competitive performance.