CodeGuard: Improving LLM Guardrails in CS Education

TL;DR

This paper introduces CodeGuard, a comprehensive framework to improve the safety of LLMs in CS education by classifying prompts, creating a large dataset, and developing a real-time detection model, significantly reducing harmful outputs.

Contribution

We propose CodeGuard, including a new taxonomy, a large prompt dataset, and PromptShield, a real-time unsafe prompt detection model, advancing LLM safety in educational settings.

Findings

PromptShield achieves 0.93 F1 score in detecting unsafe prompts.

CodeGuard reduces harmful code completions by 30-65%.

Framework improves safety without harming educational performance.

Abstract

Large language models (LLMs) are increasingly embedded in Computer Science (CS) classrooms to automate code generation, feedback, and assessment. However, their susceptibility to adversarial or ill-intentioned prompts threatens student learning and academic integrity. To cope with this important issue, we evaluate existing off-the-shelf LLMs in handling unsafe and irrelevant prompts within the domain of CS education. We identify important shortcomings in existing LLM guardrails which motivates us to propose CodeGuard, a comprehensive guardrail framework for educational AI systems. CodeGuard includes (i) a first-of-its-kind taxonomy for classifying prompts; (ii) the CodeGuard dataset, a collection of 8,000 prompts spanning the taxonomy; and (iii) PromptShield, a lightweight sentence-encoder model fine-tuned to detect unsafe prompts in real time. Experiments show that PromptShield achieves 0.93 F1 score, surpassing existing guardrail methods. Additionally, further experimentation reveals that CodeGuard reduces potentially harmful or policy-violating code completions by 30-65% without degrading performance on legitimate educational tasks. The code, datasets, and evaluation scripts are made freely available to the community.