Inner-Probe: Discovering Copyright-related Data Generation in LLM Architecture

TL;DR

Inner-Probe is a lightweight framework that effectively identifies copyrighted data influence in LLM outputs by analyzing multi-head attention results, improving detection accuracy and efficiency over existing methods.

Contribution

The paper introduces Inner-Probe, a novel approach using MHA analysis and supervised learning to determine copyrighted data influence in LLMs, addressing limitations of previous prompt-based methods.

Findings

3x improved efficiency in sub-dataset analysis

15.04% - 58.7% higher accuracy over baselines

0.104 increase in AUC for non-copyrighted data filtering

Abstract

Large Language Models (LLMs) utilize extensive knowledge databases and show powerful text generation ability. However, their reliance on high-quality copyrighted datasets raises concerns about copyright infringements in generated texts. Current research often employs prompt engineering or semantic classifiers to identify copyrighted content, but these approaches have two significant limitations: (1) Challenging to identify which specific subdataset (e.g., works from particular authors) influences an LLM's output. (2) Treating the entire training database as copyrighted, hence overlooking the inclusion of non-copyrighted training data. We propose Inner-Probe, a lightweight framework designed to evaluate the influence of copyrighted sub-datasets on LLM-generated texts. Unlike traditional methods relying solely on text, we discover that the results of multi-head attention (MHA) during LLM output generation provide more effective information. Thus, Inner-Probe performs sub-dataset contribution analysis using a lightweight LSTM based network trained on MHA results in a supervised manner. Harnessing such a prior, Inner-Probe enables non-copyrighted text detection through a concatenated global projector trained with unsupervised contrastive learning. Inner-Probe demonstrates 3x improved efficiency compared to semantic model training in sub-dataset contribution analysis on Books3, achieves 15.04% - 58.7% higher accuracy over baselines on the Pile, and delivers a 0.104 increase in AUC for non-copyrighted data filtering.