Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering

TL;DR

This paper introduces HeurVidQA, a framework that enhances video-language models for VideoQA by incorporating domain-specific heuristics, leading to improved reasoning and accuracy across multiple datasets.

Contribution

The paper presents a novel approach that uses domain-specific entity-action heuristics to refine pre-trained video-language models for better VideoQA performance.

Findings

Significant performance improvements on multiple VideoQA datasets

Effective use of domain-specific heuristics to guide model reasoning

Enhanced focus on key entities and actions improves accuracy

Abstract

Video Question Answering (VideoQA) represents a crucial intersection between video understanding and language processing, requiring both discriminative unimodal comprehension and sophisticated cross-modal interaction for accurate inference. Despite advancements in multi-modal pre-trained models and video-language foundation models, these systems often struggle with domain-specific VideoQA due to their generalized pre-training objectives. Addressing this gap necessitates bridging the divide between broad cross-modal knowledge and the specific inference demands of VideoQA tasks. To this end, we introduce HeurVidQA, a framework that leverages domain-specific entity-action heuristics to refine pre-trained video-language foundation models. Our approach treats these models as implicit knowledge engines, employing domain-specific entity-action prompters to direct the model's focus toward precise cues that enhance reasoning. By delivering fine-grained heuristics, we improve the model's ability to identify and interpret key entities and actions, thereby enhancing its reasoning capabilities. Extensive evaluations across multiple VideoQA datasets demonstrate that our method significantly outperforms existing models, underscoring the importance of integrating domain-specific knowledge into video-language models for more accurate and context-aware VideoQA.