Atomic Learning Objectives Labeling: A High-Resolution Approach for Physics Education

TL;DR

This paper presents a high-resolution atomic learning objectives system for physics education, utilizing LLMs for automated question labeling, enabling detailed mapping of cognitive processes to improve personalized learning tools.

Contribution

It introduces a novel atomic LO labeling system using LLMs for detailed cognitive process mapping in physics education, with comprehensive evaluation metrics and analysis.

Findings

LLMs can effectively automate atomic LO labeling with reasonable accuracy.

The atomic LO system provides granular insights into question content and cognitive demands.

Analysis highlights strengths and limitations of LLMs in educational labeling tasks.

Abstract

This paper introduces a novel approach to create a high-resolution "map" for physics learning: an "atomic" learning objectives (LOs) system designed to capture detailed cognitive processes and concepts required for problem solving in a college-level introductory physics course. Our method leverages Large Language Models (LLMs) for automated labeling of physics questions and introduces a comprehensive set of metrics to evaluate the quality of the labeling outcomes. The atomic LO system, covering nine chapters of an introductory physics course, uses a "subject-verb-object'' structure to represent specific cognitive processes. We apply this system to 131 questions from expert-curated question banks and the OpenStax University Physics textbook. Each question is labeled with 1-8 atomic LOs across three chapters. Through extensive experiments using various prompting strategies and LLMs, we compare automated LOs labeling results against human expert labeling. Our analysis reveals both the strengths and limitations of LLMs, providing insight into LLMs reasoning processes for labeling LOs and identifying areas for improvement in LOs system design. Our work contributes to the field of learning analytics by proposing a more granular approach to mapping learning objectives with questions. Our findings have significant implications for the development of intelligent tutoring systems and personalized learning pathways in STEM education, paving the way for more effective "learning GPS'' systems.