Training Data Attribution for Image Generation using Ontology-Aligned Knowledge Graphs

TL;DR

This paper presents a novel framework that uses ontology-aligned knowledge graphs derived from images to interpret and trace the influence of training data on generative model outputs, enhancing transparency and accountability.

Contribution

It introduces a method leveraging multimodal LLMs to extract structured, ontology-consistent representations from images for data attribution in generative models.

Findings

Effective in tracing training data influence on generated images.

Supports copyright analysis and dataset transparency.

Validated on both local and large-scale models.

Abstract

As generative models become powerful, concerns around transparency, accountability, and copyright violations have intensified. Understanding how specific training data contributes to a model's output is critical. We introduce a framework for interpreting generative outputs through the automatic construction of ontologyaligned knowledge graphs (KGs). While automatic KG construction from natural text has advanced, extracting structured and ontology-consistent representations from visual content remains challenging -- due to the richness and multi-object nature of images. Leveraging multimodal large language models (LLMs), our method extracts structured triples from images, aligned with a domain-specific ontology. By comparing the KGs of generated and training images, we can trace potential influences, enabling copyright analysis, dataset transparency, and interpretable AI. We validate our method through experiments on locally trained models via unlearning, and on large-scale models through a style-specific experiment. Our framework supports the development of AI systems that foster human collaboration, creativity and stimulate curiosity.