Token Prediction as Implicit Classification to Identify LLM-Generated Text

TL;DR

This paper presents a novel method for detecting LLM-generated text by reframing classification as a next-token prediction task, fine-tuning a T5 model for improved performance and interpretability.

Contribution

It introduces a new approach that treats LLM detection as implicit token prediction, avoiding additional classifiers and enhancing interpretability and efficiency.

Findings

Outperforms traditional classification methods in accuracy

Demonstrates high interpretability of writing style features

Creates a large dataset for LLM detection evaluation

Abstract

This paper introduces a novel approach for identifying the possible large language models (LLMs) involved in text generation. Instead of adding an additional classification layer to a base LM, we reframe the classification task as a next-token prediction task and directly fine-tune the base LM to perform it. We utilize the Text-to-Text Transfer Transformer (T5) model as the backbone for our experiments. We compared our approach to the more direct approach of utilizing hidden states for classification. Evaluation shows the exceptional performance of our method in the text classification task, highlighting its simplicity and efficiency. Furthermore, interpretability studies on the features extracted by our model reveal its ability to differentiate distinctive writing styles among various LLMs even in the absence of an explicit classifier. We also collected a dataset named OpenLLMText, containing approximately 340k text samples from human and LLMs, including GPT3.5, PaLM, LLaMA, and GPT2.