Enhancing Large Language Models for Automated Homework Assessment in Undergraduate Circuit Analysis

TL;DR

This paper improves large language models' ability to assess undergraduate circuit analysis homework by applying multi-step prompting, data augmentation, and hints, significantly increasing accuracy from 74.71% to 97.70%.

Contribution

It introduces an enhancement pipeline that significantly boosts LLM assessment accuracy for engineering homework through targeted prompting and data strategies.

Findings

Assessment accuracy improved from 74.71% to 97.70%.

Enhanced prompting addresses common errors effectively.

Method supports integration of LLMs into engineering education.

Abstract

This research full paper presents an enhancement pipeline for large language models (LLMs) in assessing homework for an undergraduate circuit analysis course, aiming to improve LLMs' capacity to provide personalized support to electrical engineering students. Existing evaluations have demonstrated that GPT-4o possesses promising capabilities in assessing student homework in this domain. Building on these findings, we enhance GPT-4o's performance through multi-step prompting, contextual data augmentation, and the incorporation of targeted hints. These strategies effectively address common errors observed in GPT-4o's responses when using simple prompts, leading to a substantial improvement in assessment accuracy. Specifically, the correct response rate for GPT-4o increases from 74.71% to 97.70% after applying the enhanced prompting and augmented data on entry-level circuit analysis topics. This work lays a foundation for the effective integration of LLMs into circuit analysis instruction and, more broadly, into engineering education.