PAS: A Training-Free Stabilizer for Temporal Encoding in Video LLMs

TL;DR

This paper introduces PAS, a simple, training-free method that stabilizes temporal encoding in Video LLMs by smoothing the attention kernel, leading to improved robustness against small frame timing shifts without additional training.

Contribution

The paper proposes PAS, a novel, training-free approach that enhances temporal stability in Video LLMs by smoothing the attention kernel, addressing a key instability issue.

Findings

PAS improves temporal stability in Video LLMs.

Experiments show consistent performance gains across benchmarks.

PAS adds negligible computational overhead.

Abstract

Video LLMs suffer from temporal inconsistency: small shifts in frame timing can flip attention and suppress relevant frames. We trace this instability to the common extension of Rotary Position Embeddings to video through multimodal RoPE. The induced inverse Fourier time kernel exhibits frame-scale ripples that multiply adjacent frames by different factors, which perturbs attention that should otherwise be governed by the raw query key inner product. We present Phase Aggregated Smoothing (PAS), a simple, training-free mechanism that applies small opposed phase offsets across heads and then aggregates their outputs. PAS preserves the per-head spectrum magnitude, while the aggregation effectively smooths the temporal kernel and reduces phase sensitivity without changing the positional encoding structure. Our analysis shows that the RoPE rotated logit can be approximated as a content dot product scaled by a time kernel; smoothing this kernel yields Lipschitz stability of attention to small temporal shifts; multi phase averaging attenuates high frequency ripples while preserving per-head spectra under Nyquist-valid sampling. Experiments on multiple video understanding benchmarks under matched token budgets show consistent improvements with negligible computational overhead. PAS provides a plug and play upgrade for robust temporal encoding in Video LLMs.