Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models

TL;DR

Video-Panda introduces a lightweight, encoder-free model for video-language understanding that significantly reduces computational costs while maintaining or improving performance on key benchmarks.

Contribution

The paper proposes a novel Spatio-Temporal Alignment Block (STAB) that processes videos without pre-trained encoders, reducing parameters by over 6.5 times and increasing processing speed.

Findings

Achieves comparable or better performance than encoder-based models on video question answering.

Outperforms Video-ChatGPT and Video-LLaVA in accuracy and temporal understanding.

Runs 3-4 times faster than previous methods.

Abstract

We present an efficient encoder-free approach for video-language understanding that achieves competitive performance while significantly reducing computational overhead. Current video-language models typically rely on heavyweight image encoders (300M-1.1B parameters) or video encoders (1B-1.4B parameters), creating a substantial computational burden when processing multi-frame videos. Our method introduces a novel Spatio-Temporal Alignment Block (STAB) that directly processes video inputs without requiring pre-trained encoders while using only 45M parameters for visual processing - at least a 6.5$\times$ reduction compared to traditional approaches. The STAB architecture combines Local Spatio-Temporal Encoding for fine-grained feature extraction, efficient spatial downsampling through learned attention and separate mechanisms for modeling frame-level and video-level relationships. Our model achieves comparable or superior performance to encoder-based approaches for open-ended video question answering on standard benchmarks. The fine-grained video question-answering evaluation demonstrates our model's effectiveness, outperforming the encoder-based approaches Video-ChatGPT and Video-LLaVA in key aspects like correctness and temporal understanding. Extensive ablation studies validate our architectural choices and demonstrate the effectiveness of our spatio-temporal modeling approach while achieving 3-4$\times$ faster processing speeds than previous methods. Code is available at https://jh-yi.github.io/Video-Panda.