Do LLMs Know to Respect Copyright Notice?

TL;DR

This paper investigates whether large language models respect copyright notices in user inputs, highlighting potential risks of copyright infringement and providing a benchmark dataset for future evaluation and alignment efforts.

Contribution

It introduces a comprehensive study on LLMs' respect for copyright notices and releases a benchmark dataset to evaluate infringement behaviors.

Findings

LLMs sometimes generate content that infringes copyright.

The study provides a conservative assessment of copyright infringement risk.

A benchmark dataset for evaluating LLMs' respect for copyright is released.

Abstract

Prior study shows that LLMs sometimes generate content that violates copyright. In this paper, we study another important yet underexplored problem, i.e., will LLMs respect copyright information in user input, and behave accordingly? The research problem is critical, as a negative answer would imply that LLMs will become the primary facilitator and accelerator of copyright infringement behavior. We conducted a series of experiments using a diverse set of language models, user prompts, and copyrighted materials, including books, news articles, API documentation, and movie scripts. Our study offers a conservative evaluation of the extent to which language models may infringe upon copyrights when processing user input containing protected material. This research emphasizes the need for further investigation and the importance of ensuring LLMs respect copyright regulations when handling user input to prevent unauthorized use or reproduction of protected content. We also release a benchmark dataset serving as a test bed for evaluating infringement behaviors by LLMs and stress the need for future alignment.