Unsupervised Model Tree Heritage Recovery

TL;DR

This paper introduces an unsupervised method to recover the hierarchical heritage of neural network models by analyzing their weights, enabling the construction of Model Trees to clarify model lineage and address intellectual property concerns.

Contribution

We propose a novel unsupervised approach to reconstruct Model Trees from neural network weights, revealing model lineage without relying on metadata or labels.

Findings

Successfully reconstructs complex Model Trees from weights

Identifies properties of weights that encode model relationships

Formulates the problem as finding a directed minimal spanning tree

Abstract

The number of models shared online has recently skyrocketed, with over one million public models available on Hugging Face. Sharing models allows other users to build on existing models, using them as initialization for fine-tuning, improving accuracy, and saving compute and energy. However, it also raises important intellectual property issues, as fine-tuning may violate the license terms of the original model or that of its training data. A Model Tree, i.e., a tree data structure rooted at a foundation model and having directed edges between a parent model and other models directly fine-tuned from it (children), would settle such disputes by making the model heritage explicit. Unfortunately, current models are not well documented, with most model metadata (e.g., "model cards") not providing accurate information about heritage. In this paper, we introduce the task of Unsupervised Model Tree Heritage Recovery (Unsupervised MoTHer Recovery) for collections of neural networks. For each pair of models, this task requires: i) determining if they are directly related, and ii) establishing the direction of the relationship. Our hypothesis is that model weights encode this information, the challenge is to decode the underlying tree structure given the weights. We discover several properties of model weights that allow us to perform this task. By using these properties, we formulate the MoTHer Recovery task as finding a directed minimal spanning tree. In extensive experiments we demonstrate that our method successfully reconstructs complex Model Trees.