Investigating Training Data Detection in AI Coders

TL;DR

This paper conducts a comprehensive empirical evaluation of training data detection methods for code large language models, introducing a new benchmark dataset and mutation strategies to assess robustness and guide future improvements.

Contribution

It introduces CodeSnitch, a function-level benchmark dataset for code data detection, and systematically evaluates seven state-of-the-art TDD methods across multiple models and mutation scenarios.

Findings

Current TDD methods have varying effectiveness on code data.

Mutation strategies reveal robustness challenges in existing TDD techniques.

The study offers insights for developing more reliable code data detection methods.

Abstract

Recent advances in code large language models (CodeLLMs) have made them indispensable tools in modern software engineering. However, these models occasionally produce outputs that contain proprietary or sensitive code snippets, raising concerns about potential non-compliant use of training data, and posing risks to privacy and intellectual property. To ensure responsible and compliant deployment of CodeLLMs, training data detection (TDD) has become a critical task. While recent TDD methods have shown promise in natural language settings, their effectiveness on code data remains largely underexplored. This gap is particularly important given code's structured syntax and distinct similarity criteria compared to natural language. To address this, we conduct a comprehensive empirical study of seven state-of-the-art TDD methods on source code data, evaluating their performance across eight CodeLLMs. To support this evaluation, we introduce CodeSnitch, a function-level benchmark dataset comprising 9,000 code samples in three programming languages, each explicitly labeled as either included or excluded from CodeLLM training. Beyond evaluation on the original CodeSnitch, we design targeted mutation strategies to test the robustness of TDD methods under three distinct settings. These mutation strategies are grounded in the well-established Type-1 to Type-4 code clone detection taxonomy. Our study provides a systematic assessment of current TDD techniques for code and offers insights to guide the development of more effective and robust detection methods in the future.