DivScore: Zero-Shot Detection of LLM-Generated Text in Specialized Domains

TL;DR

DivScore is a novel zero-shot detection method that effectively identifies LLM-generated text in specialized domains like medicine and law, outperforming existing detectors especially under domain shift and adversarial conditions.

Contribution

We introduce DivScore, a zero-shot detection framework utilizing normalized entropy scoring and domain knowledge distillation, along with a new benchmark for medical and legal text detection.

Findings

DivScore achieves 14.4% higher AUROC than state-of-the-art detectors.

DivScore attains 64.0% higher recall at a 0.1% false positive rate.

DivScore shows 22.8% advantage in AUROC in adversarial settings.

Abstract

Detecting LLM-generated text in specialized and high-stakes domains like medicine and law is crucial for combating misinformation and ensuring authenticity. However, current zero-shot detectors, while effective on general text, often fail when applied to specialized content due to domain shift. We provide a theoretical analysis showing this failure is fundamentally linked to the KL divergence between human, detector, and source text distributions. To address this, we propose DivScore, a zero-shot detection framework using normalized entropy-based scoring and domain knowledge distillation to robustly identify LLM-generated text in specialized domains. We also release a domain-specific benchmark for LLM-generated text detection in the medical and legal domains. Experiments on our benchmark show that DivScore consistently outperforms state-of-the-art detectors, with 14.4% higher AUROC and 64.0% higher recall (0.1% false positive rate threshold). In adversarial settings, DivScore demonstrates superior robustness than other baselines, achieving on average 22.8% advantage in AUROC and 29.5% in recall. Code and data are publicly available.