Mitigating Trojanized Prompt Chains in Educational LLM Use Cases: Experimental Findings and Detection Tool Design

TL;DR

This paper investigates vulnerabilities in educational LLMs where students Trojanize prompts to bypass safety measures, and introduces TrojanPromptGuard, a tool to detect and mitigate such risks, based on systematic experiments with GPT models.

Contribution

It provides the first systematic analysis of Trojan prompt vulnerabilities in educational LLM use cases and introduces a prototype detection tool, TrojanPromptGuard.

Findings

Identified key vulnerabilities in GPT-3.5 and GPT-4 to Trojanized prompts.

Developed and tested TrojanPromptGuard for automatic detection.

Provided insights for safer deployment of LLMs in education.

Abstract

The integration of Large Language Models (LLMs) in K--12 education offers both transformative opportunities and emerging risks. This study explores how students may Trojanize prompts to elicit unsafe or unintended outputs from LLMs, bypassing established content moderation systems with safety guardrils. Through a systematic experiment involving simulated K--12 queries and multi-turn dialogues, we expose key vulnerabilities in GPT-3.5 and GPT-4. This paper presents our experimental design, detailed findings, and a prototype tool, TrojanPromptGuard (TPG), to automatically detect and mitigate Trojanized educational prompts. These insights aim to inform both AI safety researchers and educational technologists on the safe deployment of LLMs for educators.