A Bayesian Approach to Harnessing the Power of LLMs in Authorship Attribution

TL;DR

This paper demonstrates that pre-trained large language models, combined with Bayesian methods, can effectively perform one-shot authorship attribution with high accuracy, offering a new approach for forensic linguistics.

Contribution

It introduces a Bayesian framework leveraging LLM probability outputs for one-shot authorship attribution, setting new performance benchmarks.

Findings

Achieved 85% accuracy on IMDb and blog datasets

Validated the approach with extensive ablation studies

Set new baselines for LLM-based authorship analysis

Abstract

Authorship attribution aims to identify the origin or author of a document. Traditional approaches have heavily relied on manual features and fail to capture long-range correlations, limiting their effectiveness. Recent advancements leverage text embeddings from pre-trained language models, which require significant fine-tuning on labeled data, posing challenges in data dependency and limited interpretability. Large Language Models (LLMs), with their deep reasoning capabilities and ability to maintain long-range textual associations, offer a promising alternative. This study explores the potential of pre-trained LLMs in one-shot authorship attribution, specifically utilizing Bayesian approaches and probability outputs of LLMs. Our methodology calculates the probability that a text entails previous writings of an author, reflecting a more nuanced understanding of authorship. By utilizing only pre-trained models such as Llama-3-70B, our results on the IMDb and blog datasets show an impressive 85\% accuracy in one-shot authorship classification across ten authors. Our findings set new baselines for one-shot authorship analysis using LLMs and expand the application scope of these models in forensic linguistics. This work also includes extensive ablation studies to validate our approach.