CL-VISTA: Benchmarking Continual Learning in Video Large Language Models

TL;DR

CL-VISTA introduces a comprehensive benchmark for evaluating continual learning in Video Large Language Models, addressing existing limitations by including diverse tasks and evaluation protocols to better assess model robustness and efficiency.

Contribution

The paper presents CL-VISTA, a new benchmark with diverse tasks and evaluation protocols specifically designed for continual learning in Video-LLMs, highlighting current trade-offs among methods.

Findings

No single continual learning method excels across all evaluated dimensions.

Methods reducing catastrophic forgetting often compromise generalization or increase resource costs.

Benchmarking reveals fundamental trade-offs in current continual learning approaches.

Abstract

Video Large Language Models (Video-LLMs) require continual learning to adapt to non-stationary real-world data. However, existing benchmarks fall short of evaluating modern foundation models: many still rely on models without large-scale pre-training, and prevailing benchmarks typically partition a single dataset into sub-tasks, resulting in high task redundancy and negligible forgetting on pre-trained Video-LLMs. To address these limitations, we propose CL-VISTA, a benchmark tailored for continual video understanding of Video-LLMs. By curating 8 diverse tasks spanning perception, understanding, and reasoning, CL-VISTA induces substantial distribution shifts that effectively expose catastrophic forgetting. To systematically assess CL methods, we establish a comprehensive evaluation framework comprising 6 distinct protocols across 3 critical dimensions: performance, computational efficiency, and memory footprint. Notably, the performance dimension incorporates a general video understanding assessment to assess whether CL methods genuinely enhance foundational intelligence or merely induce task-specific overfitting. Extensive benchmarking of 10 mainstream CL methods reveals a fundamental trade-off: no single approach achieves universal superiority across all dimensions. Methods that successfully mitigate catastrophic forgetting tend to compromise generalization or incur prohibitive computational and memory overheads. We hope CL-VISTA provides critical insights for advancing continual learning in multimodal foundation models.