LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs

TL;DR

LiteFrame introduces an efficient video encoder for Video LLMs, significantly reducing latency and enabling longer video processing without sacrificing accuracy.

Contribution

The paper proposes LiteFrame, a novel efficient video encoder backbone trained with Compressed Token Distillation to improve latency and accuracy in Video LLMs.

Findings

35% reduction in end-to-end latency

Processes 8× more frames with the same compute

Improves average video understanding accuracy

Abstract

The fundamental challenge in scaling Video Large Language Models (Video LLMs) to long-form video lies in managing the explosion of visual-token context length. Existing strategies predominantly focus on "post-hoc" token reduction -- reducing visual tokens after feature extraction to alleviate the LLM's computational overhead. While these methods effectively reduce the number of visual tokens, we observe that the primary latency bottleneck then shifts from the LLM to the expensive per-frame processing of the vision encoder. To address this, we introduce LiteFrame, a strong, yet highly efficient video encoder backbone for Video LLMs. To train LiteFrame, we propose Compressed Token Distillation (CTD), a novel training framework that teaches a compact student vision encoder to directly predict information-dense, spatio-temporally compressed representations produced by a large teacher vision model, effectively bypassing redundant computation. When coupled with further Language Model Adaptation (LMA), this approach results in a new latency-accuracy Pareto frontier -- compared with InternVL3-8B, LiteFrame provides a 35% reduction in end-to-end latency while processing 8$\times$ more frames and improves average video understanding accuracy across multiple benchmarks. Our results demonstrate a new potential path to unlocking longer-form video understanding under fixed compute budgets.