DIS-CO: Discovering Copyrighted Content in VLMs Training Data

TL;DR

DIS-CO is a novel method that detects copyrighted content in vision-language models by querying them with specific frames and analyzing their responses, revealing widespread exposure to copyrighted material.

Contribution

The paper introduces DIS-CO, a new approach for identifying copyrighted content in VLMs, along with MovieTection, a benchmark dataset for evaluation.

Findings

DIS-CO nearly doubles detection accuracy over prior methods.

All tested models show some exposure to copyrighted content.

DIS-CO effectively infers training data content without direct access.

Abstract

How can we verify whether copyrighted content was used to train a large vision-language model (VLM) without direct access to its training data? Motivated by the hypothesis that a VLM is able to recognize images from its training corpus, we propose DIS-CO, a novel approach to infer the inclusion of copyrighted content during the model's development. By repeatedly querying a VLM with specific frames from targeted copyrighted material, DIS-CO extracts the content's identity through free-form text completions. To assess its effectiveness, we introduce MovieTection, a benchmark comprising 14,000 frames paired with detailed captions, drawn from films released both before and after a model's training cutoff. Our results show that DIS-CO significantly improves detection performance, nearly doubling the average AUC of the best prior method on models with logits available. Our findings also highlight a broader concern: all tested models appear to have been exposed to some extent to copyrighted content. Our code and data are available at https://github.com/avduarte333/DIS-CO