Can We Trust LLM Detectors?

TL;DR

This paper evaluates the robustness of existing LLM detectors, revealing their brittleness under distribution shifts and proposing a supervised contrastive learning framework to improve domain generalization.

Contribution

It systematically assesses current detection methods and introduces a novel supervised contrastive learning approach for more reliable AI text detection across domains.

Findings

Supervised detectors perform well in-domain but poorly out-of-domain.

Training-free methods are highly sensitive to proxy choice.

Fundamental challenges exist in creating domain-agnostic detectors.

Abstract

The rapid adoption of LLMs has increased the need for reliable AI text detection, yet existing detectors often fail outside controlled benchmarks. We systematically evaluate 2 dominant paradigms (training-free and supervised) and show that both are brittle under distribution shift, unseen generators, and simple stylistic perturbations. To address these limitations, we propose a supervised contrastive learning (SCL) framework that learns discriminative style embeddings. Experiments show that while supervised detectors excel in-domain, they degrade sharply out-of-domain, and training-free methods remain highly sensitive to proxy choice. Overall, our results expose fundamental challenges in building domain-agnostic detectors. Our code is available at: https://github.com/HARSHITJAIS14/DetectAI