How Much Do Code Language Models Remember? An Investigation on Data Extraction Attacks before and after Fine-tuning

TL;DR

This paper investigates how much sensitive data code language models can memorize and extract, comparing pre-trained and fine-tuned models, revealing that fine-tuning reduces data extractability but can increase vulnerability in smaller models.

Contribution

The study introduces a benchmark for assessing data extraction vulnerability in both pre-trained and fine-tuned code models, providing new insights into data memorization and privacy risks.

Findings

54.9% of pre-training data can be extracted from StarCoder2-15B

Fine-tuning reduces data extractability from 54.9% to 23.5%

Data carriers and licensing info are most memorized, with some forgetting after fine-tuning

Abstract

Code language models, while widely popular, are often trained on unsanitized source code gathered from across the Internet. Previous work revealed that pre-trained models can remember the content of their training data and regurgitate them through data extraction attacks. Due to the large size of current models, only a few entities have the resources for pre-training such models. However, fine-tuning requires fewer resources and is increasingly used by both small and large entities for its effectiveness on specialized data. Such small curated data for fine-tuning might contain sensitive information or proprietary assets. In this study, we attack both pre-trained and fine-tuned code language models to investigate the extent of data extractability. We first develop a custom benchmark to assess the vulnerability of both pre-training and fine-tuning samples to extraction attacks. Our findings reveal that 54.9% of extractable pre-training data could be retrieved from StarCoder2-15B, whereas this number decreased to 23.5% after fine-tuning. This indicates that fine-tuning reduces the extractability of pre-training data. However, compared to larger models, fine-tuning smaller models increases their vulnerability to data extraction attacks on fine-tuning data. Given the potential sensitivity of fine-tuning data, this can lead to more severe consequences. Lastly, we also manually analyzed 2000 extractable samples before and after fine-tuning. We also found that data carriers and licensing information are the most likely data categories to be memorized from pre-trained and fine-tuned models, while the latter is the most likely to be forgotten after fine-tuning.