Beyond Factual QA: Mentorship-Oriented Question Answering over Long-Form Multilingual Content

TL;DR

This paper introduces MentorQA, a multilingual dataset and evaluation framework for mentorship-oriented question answering over long-form videos, emphasizing qualities like clarity and learning value beyond factual correctness.

Contribution

It presents the first multilingual dataset and evaluation framework for mentorship-focused QA from long-form videos, and compares different QA architectures in this context.

Findings

Multi-Agent pipelines outperform others in mentorship response quality.

Complex topics and low-resource languages benefit most from Multi-Agent approaches.

Automated LLM-based evaluation shows significant variation compared to human judgments.

Abstract

Question answering systems are typically evaluated on factual correctness, yet many real-world applications-such as education and career guidance-require mentorship: responses that provide reflection and guidance. Existing QA benchmarks rarely capture this distinction, particularly in multilingual and long-form settings. We introduce MentorQA, the first multilingual dataset and evaluation framework for mentorship-focused question answering from long-form videos, comprising nearly 9,000 QA pairs from 180 hours of content across four languages. We define mentorship-focused evaluation dimensions that go beyond factual accuracy, capturing clarity, alignment, and learning value. Using MentorQA, we compare Single-Agent, Dual-Agent, RAG, and Multi-Agent QA architectures under controlled conditions. Multi-Agent pipelines consistently produce higher-quality mentorship responses, with especially strong gains for complex topics and lower-resource languages. We further analyze the reliability of automated LLM-based evaluation, observing substantial variation in alignment with human judgments. Overall, this work establishes mentorship-focused QA as a distinct research problem and provides a multilingual benchmark for studying agentic architectures and evaluation design in educational AI. The dataset and evaluation framework are released at https://github.com/AIM-SCU/MentorQA.