Neural Authorship Attribution: Stylometric Analysis on Large Language Models

TL;DR

This paper investigates neural authorship attribution for AI-generated texts from large language models, analyzing stylometric features to distinguish between proprietary and open-source models, and providing empirical insights for forensic applications.

Contribution

It introduces a comprehensive empirical analysis of LLM writing signatures using stylometric features, enhancing understanding of neural authorship attribution for both proprietary and open-source models.

Findings

Stylometric features can effectively differentiate LLMs

Distinct signatures exist between proprietary and open-source models

Results support improved forensic detection of AI-generated text

Abstract

Large language models (LLMs) such as GPT-4, PaLM, and Llama have significantly propelled the generation of AI-crafted text. With rising concerns about their potential misuse, there is a pressing need for AI-generated-text forensics. Neural authorship attribution is a forensic effort, seeking to trace AI-generated text back to its originating LLM. The LLM landscape can be divided into two primary categories: proprietary and open-source. In this work, we delve into these emerging categories of LLMs, focusing on the nuances of neural authorship attribution. To enrich our understanding, we carry out an empirical analysis of LLM writing signatures, highlighting the contrasts between proprietary and open-source models, and scrutinizing variations within each group. By integrating stylometric features across lexical, syntactic, and structural aspects of language, we explore their potential to yield interpretable results and augment pre-trained language model-based classifiers utilized in neural authorship attribution. Our findings, based on a range of state-of-the-art LLMs, provide empirical insights into neural authorship attribution, paving the way for future investigations aimed at mitigating the threats posed by AI-generated misinformation.