From Course to Skill: Evaluating LLM Performance in Curricular Analytics

TL;DR

This study evaluates the effectiveness of large language models in curricular analytics, specifically for skill extraction from diverse curriculum documents, highlighting the strengths of retrieval-augmented generation over other methods.

Contribution

It systematically compares LLM-based and traditional NLP methods for skill extraction in curricular analytics, demonstrating RAG's superior performance across document types.

Findings

RAG outperforms other strategies in skill extraction tasks.

Zero-shot prompting generally underperforms compared to traditional NLP methods.

LLM performance varies significantly with model choice and prompting strategy.

Abstract

Curricular analytics (CA) -- systematic analysis of curricula data to inform program and course refinement -- becomes an increasingly valuable tool to help institutions align academic offerings with evolving societal and economic demands. Large language models (LLMs) are promising for handling large-scale, unstructured curriculum data, but it remains uncertain how reliably LLMs can perform CA tasks. In this paper, we systematically evaluate four text alignment strategies based on LLMs or traditional NLP methods for skill extraction, a core task in CA. Using a stratified sample of 400 curriculum documents of different types and a human-LLM collaborative evaluation framework, we find that retrieval-augmented generation (RAG) is the top-performing strategy across all types of curriculum documents, while zero-shot prompting performs worse than traditional NLP methods in most cases. Our findings highlight the promise of LLMs in analyzing brief and abstract curriculum documents, but also reveal that their performance can vary significantly depending on model selection and prompting strategies. This underscores the importance of carefully evaluating the performance of LLM-based strategies before large-scale deployment.