Zero-Shot Attribution for Large Language Models: A Distribution Testing Approach

TL;DR

This paper introduces Anubis, a zero-shot tool for attributing code samples to specific large language models by framing the problem as distribution testing, achieving high accuracy with limited samples.

Contribution

The paper presents a novel distribution testing approach for zero-shot attribution of code generated by LLMs, leveraging density estimates and hypothesis testing.

Findings

Achieves AUROC ≥ 0.9 in distinguishing LLMs with ~2000 samples

Uses distribution testing to attribute code samples accurately

Works effectively with limited data and common LLM access methods

Abstract

A growing fraction of all code is sampled from Large Language Models (LLMs). We investigate the problem of attributing code generated by language models using hypothesis testing to leverage established techniques and guarantees. Given a set of samples $S$ and a suspect model $\mathcal{L}^*$, our goal is to assess the likelihood of $S$ originating from $\mathcal{L}^*$. Due to the curse of dimensionality, this is intractable when only samples from the LLM are given: to circumvent this, we use both samples and density estimates from the LLM, a form of access commonly available. We introduce $\mathsf{Anubis}$, a zero-shot attribution tool that frames attribution as a distribution testing problem. Our experiments on a benchmark of code samples show that $\mathsf{Anubis}$ achieves high AUROC scores ( $\ge0.9$) when distinguishing between LLMs like DeepSeek-Coder, CodeGemma, and Stable-Code using only $\approx 2000$ samples.