LLM-Assisted Cheating Detection in Korean Language via Keystrokes

TL;DR

This study develops a keystroke-based detection framework for identifying LLM-assisted cheating in Korean writing tasks, considering cognitive context and response types, and demonstrates the effectiveness of temporal and rhythmic features.

Contribution

It introduces a novel dataset and features for detecting LLM-assisted cheating in Korean, addressing language coverage and cognitive factors not explored in prior research.

Findings

Temporal features excel in Cognition-Aware scenarios.

Rhythmic features generalize better across cognitive contexts.

Models outperform humans in detecting paraphrased and transcribed responses.

Abstract

This paper presents a keystroke-based framework for detecting LLM-assisted cheating in Korean, addressing key gaps in prior research regarding language coverage, cognitive context, and the granularity of LLM involvement. Our proposed dataset includes 69 participants who completed writing tasks under three conditions: Bona fide writing, paraphrasing ChatGPT responses, and transcribing ChatGPT responses. Each task spans six cognitive processes defined in Bloom's Taxonomy (remember, understand, apply, analyze, evaluate, and create). We extract interpretable temporal and rhythmic features and evaluate multiple classifiers under both Cognition-Aware and Cognition-Unaware settings. Temporal features perform well under Cognition-Aware evaluation scenarios, while rhythmic features generalize better under cross-cognition scenarios. Moreover, detecting bona fide and transcribed responses was easier than paraphrased ones for both the proposed models and human evaluators, with the models significantly outperforming the humans. Our findings affirm that keystroke dynamics facilitate reliable detection of LLM-assisted writing across varying cognitive demands and writing strategies, including paraphrasing and transcribing LLM-generated responses.