Origin Tracing and Detecting of LLMs

TL;DR

This paper introduces a novel, generalizable method for tracing the origin of texts generated by large language models, effective under various settings and requiring limited data, with extensive experiments and open resources.

Contribution

The paper presents an innovative contrastive feature-based algorithm for LLM origin tracing that works in both white-box and black-box scenarios, requiring minimal data and supporting new model detection.

Findings

The method effectively traces LLM origins in diverse settings.

Limited data suffices for accurate origin detection.

The approach reveals insights into AI-generated text similarities and detection challenges.

Abstract

The extraordinary performance of large language models (LLMs) heightens the importance of detecting whether the context is generated by an AI system. More importantly, while more and more companies and institutions release their LLMs, the origin can be hard to trace. Since LLMs are heading towards the time of AGI, similar to the origin tracing in anthropology, it is of great importance to trace the origin of LLMs. In this paper, we first raise the concern of the origin tracing of LLMs and propose an effective method to trace and detect AI-generated contexts. We introduce a novel algorithm that leverages the contrastive features between LLMs and extracts model-wise features to trace the text origins. Our proposed method works under both white-box and black-box settings therefore can be widely generalized to detect various LLMs.(e.g. can be generalized to detect GPT-3 models without the GPT-3 models). Also, our proposed method requires only limited data compared with the supervised learning methods and can be extended to trace new-coming model origins. We construct extensive experiments to examine whether we can trace the origins of given texts. We provide valuable observations based on the experimental results, such as the difficulty level of AI origin tracing, and the AI origin similarities, and call for ethical concerns of LLM providers. We are releasing all codes and data as a toolkit and benchmark for future AI origin tracing and detecting studies. \footnote{We are releasing all available resource at \url{https://github.com/OpenLMLab/}.}