MovieCORE: COgnitive REasoning in Movies

TL;DR

MovieCORE introduces a challenging VQA dataset focused on deep cognitive understanding of movies, utilizing LLMs for question generation and an enhancement module to improve reasoning in AI models.

Contribution

The paper presents MovieCORE, a novel dataset emphasizing deep cognitive questions, and an agentic approach with LLMs and ACE to enhance video reasoning capabilities.

Findings

ACE improves reasoning performance by up to 25%.

Dataset quality assessed through cognitive tests.

Enhanced understanding of model limitations on nuanced movie questions.

Abstract

This paper introduces MovieCORE, a novel video question answering (VQA) dataset designed to probe deeper cognitive understanding of movie content. Unlike existing datasets that focus on surface-level comprehension, MovieCORE emphasizes questions that engage System-2 thinking while remaining specific to the video material. We present an innovative agentic brainstorming approach, utilizing multiple large language models (LLMs) as thought agents to generate and refine high-quality question-answer pairs. To evaluate dataset quality, we develop a set of cognitive tests assessing depth, thought-provocation potential, and syntactic complexity. We also propose a comprehensive evaluation scheme for assessing VQA model performance on deeper cognitive tasks. To address the limitations of existing video-language models (VLMs), we introduce an agentic enhancement module, Agentic Choice Enhancement (ACE), which improves model reasoning capabilities post-training by up to 25%. Our work contributes to advancing movie understanding in AI systems and provides valuable insights into the capabilities and limitations of current VQA models when faced with more challenging, nuanced questions about cinematic content. Our project page, dataset and code can be found at https://joslefaure.github.io/assets/html/moviecore.html.