Uncovering Pretraining Code in LLMs: A Syntax-Aware Attribution Approach

TL;DR

This paper introduces SynPrune, a syntax-aware method for detecting whether specific code was part of an LLM's training data, improving accuracy by leveraging programming language structure.

Contribution

SynPrune is the first syntax-aware membership inference attack for code, outperforming prior methods by utilizing programming language syntax to improve detection accuracy.

Findings

SynPrune outperforms existing MIAs in code detection accuracy.

The method is robust across different code lengths and syntax categories.

SynPrune effectively excludes syntactically required tokens from attribution.

Abstract

As large language models (LLMs) become increasingly capable, concerns over the unauthorized use of copyrighted and licensed content in their training data have grown, especially in the context of code. Open-source code, often protected by open source licenses (e.g, GPL), poses legal and ethical challenges when used in pretraining. Detecting whether specific code samples were included in LLM training data is thus critical for transparency, accountability, and copyright compliance. We propose SynPrune, a syntax-pruned membership inference attack method tailored for code. Unlike prior MIA approaches that treat code as plain text, SynPrune leverages the structured and rule-governed nature of programming languages. Specifically, it identifies and excludes consequent tokens that are syntactically required and not reflective of authorship, from attribution when computing membership scores. Experimental results show that SynPrune consistently outperforms the state-of-the-arts. Our method is also robust across varying function lengths and syntax categories.