SPOT: Text Source Prediction from Originality Score Thresholding

TL;DR

SPOT is a novel method that classifies whether a text was generated by an LLM or a human by analyzing originality scores, demonstrating robustness across various models and data conditions.

Contribution

This paper introduces SPOT, a new approach for source prediction of texts based on originality scores, focusing on trust rather than validity detection.

Findings

SPOT effectively distinguishes LLM-generated text from human text.

The method is robust across different LLM architectures and training data.

It performs well even with compressed models and varied evaluation datasets.

Abstract

The wide acceptance of large language models (LLMs) has unlocked new applications and social risks. Popular countermeasures aim at detecting misinformation, usually involve domain specific models trained to recognize the relevance of any information. Instead of evaluating the validity of the information, we propose to investigate LLM generated text from the perspective of trust. In this study, we define trust as the ability to know if an input text was generated by a LLM or a human. To do so, we design SPOT, an efficient method, that classifies the source of any, standalone, text input based on originality score. This score is derived from the prediction of a given LLM to detect other LLMs. We empirically demonstrate the robustness of the method to the architecture, training data, evaluation data, task and compression of modern LLMs.