Open-Ended and Knowledge-Intensive Video Question Answering

TL;DR

This paper explores knowledge-intensive video question answering using retrieval-augmented generation, demonstrating that optimized retrieval strategies significantly enhance accuracy on complex, open-ended questions in video understanding tasks.

Contribution

It introduces a comprehensive analysis of retrieval-augmented methods for KI-VideoQA, highlighting the importance of modality choice, retrieval strategy, and query formulation for improved performance.

Findings

Retrieval augmentation improves KI-VideoQA performance.

Modality and retrieval method critically affect results.

Achieved 17.5% accuracy improvement on KnowIT VQA.

Abstract

Video question answering that requires external knowledge beyond the visual content remains a significant challenge in AI systems. While models can effectively answer questions based on direct visual observations, they often falter when faced with questions requiring broader contextual knowledge. To address this limitation, we investigate knowledge-intensive video question answering (KI-VideoQA) through the lens of multi-modal retrieval-augmented generation, with a particular focus on handling open-ended questions rather than just multiple-choice formats. Our comprehensive analysis examines various retrieval augmentation approaches using cutting-edge retrieval and vision language models, testing both zero-shot and fine-tuned configurations. We investigate several critical dimensions: the interplay between different information sources and modalities, strategies for integrating diverse multi-modal contexts, and the dynamics between query formulation and retrieval result utilization. Our findings reveal that while retrieval augmentation shows promise in improving model performance, its success is heavily dependent on the chosen modality and retrieval methodology. The study also highlights the critical role of query construction and retrieval depth optimization in effective knowledge integration. Through our proposed approach, we achieve a substantial 17.5% improvement in accuracy on multiple choice questions in the KnowIT VQA dataset, establishing new state-of-the-art performance levels.