"Yes, My LoRD." Guiding Language Model Extraction with Locality Reinforced Distillation

TL;DR

LoRD introduces a novel, locality-reinforced distillation approach tailored for extracting large language models, improving efficiency and effectiveness over existing methods by aligning extraction with LLM training tasks.

Contribution

The paper presents LoRD, a new model extraction algorithm for LLMs that uses policy-gradient training and theoretical analysis to enhance extraction performance and reduce query complexity.

Findings

LoRD outperforms existing extraction methods on commercial LLMs.

LoRD reduces query complexity and mitigates watermark protection.

Theoretical analysis confirms convergence and alignment with LLM training procedures.

Abstract

Model extraction attacks (MEAs) on large language models (LLMs) have received increasing attention in recent research. However, existing attack methods typically adapt the extraction strategies originally developed for deep neural networks (DNNs). They neglect the underlying inconsistency between the training tasks of MEA and LLM alignment, leading to suboptimal attack performance. To tackle this issue, we propose Locality Reinforced Distillation (LoRD), a novel model extraction algorithm specifically designed for LLMs. In particular, LoRD employs a newly defined policy-gradient-style training task that utilizes the responses of victim model as the signal to guide the crafting of preference for the local model. Theoretical analyses demonstrate that I) The convergence procedure of LoRD in model extraction is consistent with the alignment procedure of LLMs, and II) LoRD can reduce query complexity while mitigating watermark protection through our exploration-based stealing. Extensive experiments validate the superiority of our method in extracting various state-of-the-art commercial LLMs. Our code is available at: https://github.com/liangzid/LoRD-MEA .