PEMUTA: Pedagogically-Enriched Multi-Granular Undergraduate Thesis Assessment

TL;DR

PEMUTA introduces a pedagogically-enriched, multi-granular assessment framework for undergraduate theses using LLMs, guided by educational theories and hierarchical prompting, to better reflect diverse academic competencies.

Contribution

It pioneers a hierarchical, pedagogically-informed LLM framework for detailed UGTE evaluation across multiple criteria, integrating educational theories and multi-dimensional assessment techniques.

Findings

PEMUTA aligns closely with expert judgments in UGTE evaluation.

The hierarchical prompting improves assessment granularity and pedagogical relevance.

The framework demonstrates strong potential for automated, fine-grained thesis assessment.

Abstract

The undergraduate thesis (UGTE) plays an indispensable role in assessing a student's cumulative academic development throughout their college years. Although large language models (LLMs) have advanced education intelligence, they typically focus on holistic assessment with only one single evaluation score, but ignore the intricate nuances across multifaceted criteria, limiting their ability to reflect structural criteria, pedagogical objectives, and diverse academic competencies. Meanwhile, pedagogical theories have long informed manual UGTE evaluation through multi-dimensional assessment of cognitive development, disciplinary thinking, and academic performance, yet remain underutilized in automated settings. Motivated by the research gap, we pioneer PEMUTA, a pedagogically-enriched framework that effectively activates domain-specific knowledge from LLMs for multi-granular UGTE assessment. Guided by Vygotsky's theory and Bloom's Taxonomy, PEMUTA incorporates a hierarchical prompting scheme that evaluates UGTEs across six fine-grained dimensions: Structure, Logic, Originality, Writing, Proficiency, and Rigor (SLOWPR), followed by holistic synthesis. Two in-context learning techniques, \ie, few-shot prompting and role-play prompting, are also incorporated to further enhance alignment with expert judgments without fine-tuning. We curate a dataset of authentic UGTEs with expert-provided SLOWPR-aligned annotations to support multi-granular UGTE assessment. Extensive experiments demonstrate that PEMUTA achieves strong alignment with expert evaluations, and exhibits strong potential for fine-grained, pedagogically-informed UGTE evaluations.