LEMON: How Well Do MLLMs Perform Temporal Multimodal Understanding on Instructional Videos?

TL;DR

LEMON introduces a comprehensive benchmark for evaluating multimodal large language models on long-form, educational STEM videos, emphasizing temporal reasoning, cross-modal integration, and pedagogical understanding.

Contribution

This paper presents LEMON, a new benchmark with diverse tasks and rich content to assess MLLMs' capabilities in understanding and reasoning over instructional videos.

Findings

State-of-the-art MLLMs perform poorly on temporal reasoning tasks.

LEMON reveals significant gaps in current models' understanding of instructional content.

Benchmark encourages development of more advanced multimodal reasoning models.

Abstract

Recent multimodal large language models (MLLMs) have shown remarkable progress across vision, audio, and language tasks, yet their performance on long-form, knowledge-intensive, and temporally structured educational content remains largely unexplored. To bridge this gap, we introduce LEMON, a Lecture-based Evaluation benchmark for MultimOdal uNderstanding, focusing on STEM lecture videos that require long-horizon reasoning and cross-modal integration. LEMON comprises 2,277 video segments spanning 5 disciplines and 29 courses, with an average duration of 196.1 seconds, yielding 4,181 high-quality QA pairs, including 3,413 multiple-choice and 768 open-ended questions. Distinct from existing video benchmarks, LEMON features: (1) semantic richness and disciplinary density, (2) tightly coupled video-audio-text modalities, (3) explicit temporal and pedagogical structure, and (4) contextually linked multi-turn questioning. It further encompasses six major tasks and twelve subtasks, covering the full cognitive spectrum from perception to reasoning and then to generation. Comprehensive experiments reveal substantial performance gaps across tasks, highlighting that even state-of-the-art MLLMs like GPT-4o struggle with temporal reasoning and instructional prediction. We expect LEMON to serve as an extensible and challenging benchmark for advancing multimodal perception, reasoning, and generation in long-form instructional contents.