Optimizing Multimodal LLMs for Egocentric Video Understanding: A Solution for the HD-EPIC VQA Challenge

TL;DR

This paper introduces a comprehensive framework to enhance multimodal large language models for egocentric video question answering, addressing challenges like temporal reasoning and ambiguous queries, and achieves significant accuracy improvements on the HD-EPIC VQA benchmark.

Contribution

It presents a novel pipeline combining query preprocessing, domain-specific fine-tuning, Temporal Chain-of-Thought prompting, and post-processing for improved video understanding.

Findings

Achieved 41.6% accuracy on HD-EPIC VQA.

Demonstrated the effectiveness of T-CoT prompting for multi-step reasoning.

Highlighted the importance of holistic pipeline optimization.

Abstract

Multimodal Large Language Models (MLLMs) struggle with complex video QA benchmarks like HD-EPIC VQA due to ambiguous queries/options, poor long-range temporal reasoning, and non-standardized outputs. We propose a framework integrating query/choice pre-processing, domain-specific Qwen2.5-VL fine-tuning, a novel Temporal Chain-of-Thought (T-CoT) prompting for multi-step reasoning, and robust post-processing. This system achieves 41.6% accuracy on HD-EPIC VQA, highlighting the need for holistic pipeline optimization in demanding video understanding. Our code, fine-tuned models are available at https://github.com/YoungSeng/Egocentric-Co-Pilot.