HCQA @ Ego4D EgoSchema Challenge 2024

TL;DR

This paper introduces HCQA, a hierarchical approach for egocentric video question answering that combines captioning, summarization, and reasoning, achieving 75% accuracy on the EgoSchema challenge.

Contribution

The paper presents a novel Hierarchical Comprehension scheme for egocentric video QA, integrating captioning, summarization, and reasoning stages for improved performance.

Findings

Achieved 75% accuracy on EgoSchema blind test set.

Effectively captures local and global visual information.

Outperforms previous methods in egocentric QA.

Abstract

In this report, we present our champion solution for Ego4D EgoSchema Challenge in CVPR 2024. To deeply integrate the powerful egocentric captioning model and question reasoning model, we propose a novel Hierarchical Comprehension scheme for egocentric video Question Answering, named HCQA. It consists of three stages: Fine-grained Caption Generation, Context-driven Summarization, and Inference-guided Answering. Given a long-form video, HCQA captures local detailed visual information and global summarised visual information via Fine-grained Caption Generation and Context-driven Summarization, respectively. Then in Inference-guided Answering, HCQA utilizes this hierarchical information to reason and answer given question. On the EgoSchema blind test set, HCQA achieves 75% accuracy in answering over 5,000 human curated multiple-choice questions. Our code will be released at https://github.com/Hyu-Zhang/HCQA.