Pruning as a Defense: Reducing Memorization in Large Language Models

TL;DR

This paper explores how simple pruning techniques can significantly reduce memorization in large language models, thereby enhancing privacy and security by mitigating membership inference risks.

Contribution

It introduces pruning as an effective method to decrease memorization in LLMs, a novel approach for privacy preservation in large-scale models.

Findings

Pruning reduces memorization in large language models.

Pruning diminishes susceptibility to membership inference attacks.

Pruning maintains model performance while improving privacy.

Abstract

Large language models have been shown to memorize significant portions of their training data, which they can reproduce when appropriately prompted. This work investigates the impact of simple pruning techniques on this behavior. Our findings reveal that pruning effectively reduces the extent of memorization in LLMs, demonstrating its potential as a foundational approach for mitigating membership inference attacks.