Test-Time Temporal Sampling for Efficient MLLM Video Understanding

TL;DR

This paper introduces T3S, a training-free inference method that efficiently processes long videos with multimodal large language models by sampling multiple short subsequences and aggregating predictions, improving speed and accuracy.

Contribution

T3S is a novel, inference-time, plug-and-play approach that reduces computational complexity and enhances long-video understanding without model retraining or fine-tuning.

Findings

Improves accuracy by up to 3.1% on benchmarks.

Reduces inference delay by 2.04 times.

Operates with minimal integration effort.

Abstract

Processing long videos with multimodal large language models (MLLMs) poses a significant computational challenge, as the model's self-attention mechanism scales quadratically with the number of video tokens, resulting in high computational demand and slow inference speed. Current solutions, such as rule-based sub-sampling, learned frame selector, or memory-based summarization, often introduce their own trade-offs: they compromise accuracy, necessitate additional training, or decrease inference speed. In this paper, we propose Test-Time Temporal Sampling (T3S), a training-free, plug-and-play inference wrapper that enables MLLMs to process long videos both efficiently and effectively. T3S exploits spatiotemporal redundancy by generating multiple short and diverse subsequences of video tokens at inference time, packing them within a single forward pass, and aggregating their predictions. This multi-subsequence formulation broadens visual coverage while reducing the computational cost of self-attention from $O(L^2)$ to $O(\sum_{i=1}^m \alpha_i^2L^2)$, where $\sum_{i=1}^m \alpha_i^2 < 1$. Extensive experiments on long video understanding benchmarks demonstrate that T3S improves accuracy by up to 3.1% and reduces first token delay by $2.04\times$, all with minimal integration effort. Our approach operates entirely at inference time, requires no model modifications or fine-tuning, and is compatible with a wide range of pretrained MLLMs. T3S turns video redundancy into a computational advantage, offering a scalable solution for long-video understanding. The code is available at https://github.com/kaibinwang3/T3S.