How to Protect Copyright Data in Optimization of Large Language Models?

TL;DR

This paper presents a theoretical method for training large language models that prevents them from generating copyrighted data by reformulating training as a softmax regression problem with copyright protection.

Contribution

It introduces a novel approach to training LLMs that avoids copyright issues by leveraging a reformulation of the training process as a softmax regression problem.

Findings

The training of LLMs can be viewed as a softmax regression problem.

A new method efficiently prevents copyright data generation during training.

The approach offers a theoretical framework for copyright-safe LLM training.

Abstract

Large language models (LLMs) and generative AI have played a transformative role in computer research and applications. Controversy has arisen as to whether these models output copyrighted data, which can occur if the data the models are trained on is copyrighted. LLMs are built on the transformer neural network architecture, which in turn relies on a mathematical computation called Attention that uses the softmax function. In this paper, we show that large language model training and optimization can be seen as a softmax regression problem. We then establish a method of efficiently performing softmax regression, in a way that prevents the regression function from generating copyright data. This establishes a theoretical method of training large language models in a way that avoids generating copyright data.