EVQAScore: A Fine-grained Metric for Video Question Answering Data Quality Evaluation
Hao Liang, Zirong Chen, Hejun Dong, Wentao Zhang

TL;DR
EVQAScore is a novel reference-free evaluation metric for video QA and caption data quality, utilizing keyword extraction and frame sampling to improve robustness and efficiency, achieving state-of-the-art results on benchmark datasets.
Contribution
The paper introduces EVQAScore, a new evaluation method for video QA data quality that is efficient, robust, and outperforms existing metrics on standard benchmarks.
Findings
Achieves state-of-the-art correlation scores on VATEX-EVAL.
Outperforms previous methods using only 12.5% of data.
Effective in selecting high-quality data for training.
Abstract
Video question-answering (QA) is a core task in video understanding. Evaluating the quality of video QA and video caption data quality for training video large language models (VideoLLMs) is an essential challenge. Although various methods have been proposed for assessing video caption quality, there remains a lack of dedicated evaluation methods for Video QA. To address this gap, we introduce EVQAScore, a reference-free method that leverages keyword extraction to assess both video caption and video QA data quality. Additionally, we incorporate frame sampling and rescaling techniques to enhance the efficiency and robustness of our evaluation, this enables our score to evaluate the quality of extremely long videos. Our approach achieves state-of-the-art (SOTA) performance (32.8 for Kendall correlation and 42.3 for Spearman correlation, 4.7 and 5.9 higher than the previous method PAC-S++)…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Domain Adaptation and Few-Shot Learning
