Homoglyph-based Adversarial Perturbation of Introductory Computer Science Theory Problems

TL;DR

This paper introduces a homoglyph-based adversarial perturbation method to modify computer science questions, effectively challenging AI tools and highlighting vulnerabilities in automated problem-solving systems.

Contribution

It presents a novel homoglyph-based perturbation technique and an interactive tool to alter questions without changing their semantic meaning, exposing AI vulnerabilities.

Findings

Effective perturbation of theoretical CS problems demonstrated

Method preserves question semantics while altering appearance

Interactive tool facilitates easy application of the perturbation

Abstract

Different AI tools such as ChatGPT, Gemini, and Claude are becoming very popular. Although they are helpful for many day-to-day tasks, they can be used in unexpected ways. For example, the learning objectives of a course may not be achieved if students use these tools to solve their homework problems. This paper proposes a simple method to address this issue in the lazy student model. The method uses homoglyph-based adversarial perturbation to first modify the question without changing the semantic meaning of the question. Then a few characters are perturbed by their homoglyphs. Our experimental result shows the theoretical problems of introductory computer science courses can be effectively perturbed. We also propose an interactive tool to conveniently use our method.