VCEval: Rethinking What is a Good Educational Video and How to Automatically Evaluate It

TL;DR

This paper introduces VCEval, a new framework that automatically assesses the quality of educational videos by modeling the task as a question-answering problem using language models, aiming to improve evaluation consistency and interpretability.

Contribution

The paper presents a novel evaluation framework, VCEval, based on three principles, and constructs a new dataset for automatic quality assessment of educational videos.

Findings

VCEval effectively differentiates video quality levels.

The framework produces interpretable evaluation results.

The approach leverages language models for assessment.

Abstract

Online courses have significantly lowered the barrier to accessing education, yet the varying content quality of these videos poses challenges. In this work, we focus on the task of automatically evaluating the quality of video course content. We have constructed a dataset with a substantial collection of video courses and teaching materials. We propose three evaluation principles and design a new evaluation framework, \textit{VCEval}, based on these principles. The task is modeled as a multiple-choice question-answering task, with a language model serving as the evaluator. Our method effectively distinguishes video courses of different content quality and produces a range of interpretable results.