E-VRAG: Enhancing Long Video Understanding with Resource-Efficient Retrieval Augmented Generation

TL;DR

E-VRAG is a resource-efficient retrieval-augmented generation framework that improves long video understanding by reducing computational costs and enhancing accuracy through hierarchical filtering, lightweight scoring, and multi-view question answering.

Contribution

The paper introduces E-VRAG, a novel framework combining hierarchical frame filtering, lightweight VLM scoring, and a global score-based retrieval strategy for efficient long video understanding.

Findings

Achieves 70% reduction in computational cost

Outperforms baseline methods in accuracy

Operates without additional training

Abstract

Vision-Language Models (VLMs) have enabled substantial progress in video understanding by leveraging cross-modal reasoning capabilities. However, their effectiveness is limited by the restricted context window and the high computational cost required to process long videos with thousands of frames. Retrieval-augmented generation (RAG) addresses this challenge by selecting only the most relevant frames as input, thereby reducing the computational burden. Nevertheless, existing video RAG methods struggle to balance retrieval efficiency and accuracy, particularly when handling diverse and complex video content. To address these limitations, we propose E-VRAG, a novel and efficient video RAG framework for video understanding. We first apply a frame pre-filtering method based on hierarchical query decomposition to eliminate irrelevant frames, reducing computational costs at the data level. We then employ a lightweight VLM for frame scoring, further reducing computational costs at the model level. Additionally, we propose a frame retrieval strategy that leverages the global statistical distribution of inter-frame scores to mitigate the potential performance degradation from using a lightweight VLM. Finally, we introduce a multi-view question answering scheme for the retrieved frames, enhancing the VLM's capability to extract and comprehend information from long video contexts. Experiments on four public benchmarks show that E-VRAG achieves about 70% reduction in computational cost and higher accuracy compared to baseline methods, all without additional training. These results demonstrate the effectiveness of E-VRAG in improving both efficiency and accuracy for video RAG tasks.