CATS: Curvature Aware Temporal Selection for efficient long video understanding

TL;DR

CATS is a curvature-aware frame selection method that improves long video understanding by efficiently identifying salient content, balancing accuracy and computational cost.

Contribution

It introduces a novel curvature-aware approach to select informative frames, outperforming prior lightweight methods while maintaining high efficiency.

Findings

CATS outperforms prior lightweight methods like AKS on LongVideoBench and VideoMME.

CATS achieves 93-95% of MIRA's accuracy with only 3-4% of its preprocessing cost.

CATS produces more coherent and informative descriptions in LLM-based evaluation.

Abstract

Understanding long videos with multimodal large language models (MLLMs) requires selecting a small subset of informative frames under strict computational budgets, where exhaustive processing is infeasible and optimal selection is combinatorial. We propose CATS, a curvature-aware frame selection method that explicitly models the temporal geometry of query-frame relevance to identify salient events and their surrounding context. By leveraging temporal curvature to adapt selection density, CATS captures both abrupt transitions and gradually evolving content while suppressing redundant frames. Under a fixed backbone and frame budget, CATS consistently outperforms prior lightweight approaches such as AKS on LongVideoBench and VideoMME. While multi-stage methods such as MIRA achieve higher absolute accuracy, they incur substantial computational overhead; in contrast, CATS retains approximately 93-95% of MIRA's performance while requiring only 3-4% of its preprocessing cost, yielding a favorable efficiency-accuracy trade-off. Beyond answer accuracy, we evaluate description generation using an LLM-as-a-judge protocol, and the obtained results show that CATS produces more coherent and informative outputs, indicating improved grounding in visual evidence. These results position CATS as a computationally efficient and principled approach to long-video understanding.