Evaluating Software Plagiarism Detection in the Age of AI: Automated Obfuscation and Lessons for Academic Integrity

TL;DR

This study assesses the robustness of software plagiarism detection tools against advanced automated and AI-generated obfuscation techniques, revealing significant improvements but also highlighting ongoing vulnerabilities impacting academic integrity.

Contribution

It provides a comprehensive evaluation of defense mechanisms against AI-based obfuscation, expanding understanding of their effectiveness on real-world datasets and diverse attack methods.

Findings

Defense mechanisms significantly improve detection rates.

AI-generated obfuscation remains a challenge for current tools.

Enhanced detection across over four million program comparisons.

Abstract

Plagiarism in programming assignments is a persistent issue in computer science education, increasingly complicated by the emergence of automated obfuscation attacks. While software plagiarism detectors are widely used to identify suspicious similarities at scale and are resilient to simple obfuscation techniques, they are vulnerable to advanced obfuscation based on structural modification of program code that preserves the original program behavior. While different defense mechanisms have been proposed to increase resilience against these attacks, their current evaluation is limited to the scope of attacks used and lacks a comprehensive investigation regarding AI-based obfuscation. In this paper, we investigate the resilience of these defense mechanisms against a broad range of automated obfuscation attacks, including both algorithmic and AI-generated methods, and for a wide variety of real-world datasets. We evaluate the improvements of two defense mechanisms over the plagiarism detector JPlag across over four million pairwise program comparisons. Our results show significant improvements in detecting obfuscated plagiarism instances, and we observe an improved detection of AI-generated programs, even though the defense mechanisms are not designed for this use case. Based on our findings, we provide an in-depth discussion of their broader implications for academic integrity and the role of AI in education.