WIP: Leveraging LLMs for Enforcing Design Principles in Student Code: Analysis of Prompting Strategies and RAG

TL;DR

This paper investigates using Large Language Models combined with Retrieval-Augmented Generation to automate feedback on student code, focusing on design principles, and evaluates different prompting strategies to enhance code review in educational settings.

Contribution

It introduces a novel approach integrating LLMs and RAG for automated code review of design principles, analyzing prompting strategies to improve feedback accuracy.

Findings

Preliminary results show improved code quality with LLM-based feedback.

Different prompting strategies significantly affect assessment accuracy.

RAG integration enhances the system's ability to evaluate design adherence.

Abstract

This work-in-progress research-to-practice paper explores the integration of Large Language Models (LLMs) into the code-review process for open-source software projects developed in computer science and software engineering courses. The focus is on developing an automated feedback tool that evaluates student code for adherence to key object-oriented design principles, addressing the need for more effective and scalable methods to teach software design best practices. The innovative practice involves leveraging LLMs and Retrieval-Augmented Generation (RAG) to create an automated feedback system that assesses student code for principles like SOLID, DRY, and design patterns. It analyzes the effectiveness of various prompting strategies and the RAG integration. Preliminary findings show promising improvements in code quality. Future work will aim to improve model accuracy and expand support for additional design principles.