OpenForge: Probabilistic Metadata Integration

TL;DR

OpenForge is a probabilistic framework that combines large language models and Markov Random Fields to improve the accuracy and consistency of metadata relationship integration across diverse data sources.

Contribution

It introduces a novel two-stage approach for metadata integration, leveraging LLMs for priors and probabilistic graphical models for refinement, formalized as an optimization problem.

Findings

Outperforms GPT-4 by 25 F1-score points in metadata vocabulary matching.

Demonstrates effectiveness and efficiency on real-world datasets.

Captures relationship properties like transitivity probabilistically.

Abstract

Modern data stores increasingly rely on metadata for enabling diverse activities such as data cataloging and search. However, metadata curation remains a labor-intensive task, and the broader challenge of metadata maintenance -- ensuring its consistency, usefulness, and freshness -- has been largely overlooked. In this work, we tackle the problem of resolving relationships among metadata concepts from disparate sources. These relationships are critical for creating clean, consistent, and up-to-date metadata repositories, and a central challenge for metadata integration. We propose OpenForge, a two-stage prior-posterior framework for metadata integration. In the first stage, OpenForge exploits multiple methods including fine-tuned large language models to obtain prior beliefs about concept relationships. In the second stage, OpenForge refines these predictions by leveraging Markov Random Field, a probabilistic graphical model. We formalize metadata integration as an optimization problem, where the objective is to identify the relationship assignments that maximize the joint probability of assignments. The MRF formulation allows OpenForge to capture prior beliefs while encoding critical relationship properties, such as transitivity, in probabilistic inference. Experiments on real-world datasets demonstrate the effectiveness and efficiency of OpenForge. On a use case of matching two metadata vocabularies, OpenForge outperforms GPT-4, the second-best method, by 25 F1-score points.