Fooling LLM graders into giving better grades through neural activity guided adversarial prompting

TL;DR

This paper uncovers vulnerabilities in LLM-based essay grading by identifying neural activity patterns that can be exploited through adversarial prompts to artificially inflate grades, revealing inherent biases and proposing mitigation strategies.

Contribution

It introduces a systematic method to detect and exploit hidden neural biases in LLM evaluators and demonstrates how minor template changes can mitigate these biases.

Findings

Adversarial prompts can significantly increase LLM grades beyond human levels.

The attack transfers from white-box to black-box models, including commercial systems.

A specific 'magic word' influences the effectiveness of the attack, linked to chat template structures.

Abstract

The deployment of artificial intelligence (AI) in critical decision-making and evaluation processes raises concerns about inherent biases that malicious actors could exploit to distort decision outcomes. We propose a systematic method to reveal such biases in AI evaluation systems and apply it to automated essay grading as an example. Our approach first identifies hidden neural activity patterns that predict distorted decision outcomes and then optimizes an adversarial input suffix to amplify such patterns. We demonstrate that this combination can effectively fool large language model (LLM) graders into assigning much higher grades than humans would. We further show that this white-box attack transfers to black-box attacks on other models, including commercial closed-source models like Gemini. They further reveal the existence of a "magic word" that plays a pivotal role in the efficacy of the attack. We trace the origin of this magic word bias to the structure of commonly-used chat templates for supervised fine-tuning of LLMs and show that a minor change in the template can drastically reduce the bias. This work not only uncovers vulnerabilities in current LLMs but also proposes a systematic method to identify and remove hidden biases, contributing to the goal of ensuring AI safety and security.