ArtifactLinker: Linking Scientific Artifacts for Automatic State-of-the-Art Discovery

TL;DR

ArtifactLinker is a framework that leverages graph neural networks and large language models to automatically discover the best models for datasets by analyzing artifacts on platforms like HuggingFace.

Contribution

It introduces a novel two-stage approach combining ranking and verification for automatic SOTA discovery using artifact graphs and LLMs.

Findings

Graph structures effectively predict missing links between artifacts.

ArtifactLinker successfully discovers potential SOTA models and insights.

The ArtifactBench benchmark enables evaluation of artifact link prediction methods.

Abstract

Scientific artifacts such as models and datasets are foundations for research. With the rapid growth of platforms like HuggingFace, researchers now have access to a large number of artifacts. Yet, a key challenge remains: how can we automatically discover the state-of-the-art (SOTA) model for a given dataset by fully leveraging existing artifacts? We formalize this task as automatic SOTA discovery by modeling HuggingFace as an artifact graph, where nodes are models/datasets and edges represent evaluations. We propose ArtifactLinker, a two-stage framework: (1) ranking promising unobserved model--dataset links using Graph Neural Networks (GNNs) or graph-augmented Large Language Models (LLMs), and (2) verifying top-ranked links via coding experiments with LLM-based agents. We further introduce a benchmark named ArtifactBench with 14,053 artifacts and 51,337 relations to evaluate the performance of both stages. Results show that (1) graph structures between existing artifacts are effective for missing link prediction; (2) end-to-end ranking and verification with ArtifactLinker help discover potential SOTA results and research insights.