Computing Multi-Relational Sufficient Statistics for Large Databases

TL;DR

This paper introduces a scalable dynamic programming algorithm for computing multi-relational sufficient statistics, including negative relationships, in large databases without materializing join tables, enabling advanced statistical analysis.

Contribution

It presents a novel M"obius virtual join algorithm that efficiently computes counts involving positive and negative relationships in large, complex databases.

Findings

Scales to datasets over 1 million tuples

Improves feature selection and rule mining

Enables Bayesian network learning with relational data

Abstract

Databases contain information about which relationships do and do not hold among entities. To make this information accessible for statistical analysis requires computing sufficient statistics that combine information from different database tables. Such statistics may involve any number of {\em positive and negative} relationships. With a naive enumeration approach, computing sufficient statistics for negative relationships is feasible only for small databases. We solve this problem with a new dynamic programming algorithm that performs a virtual join, where the requisite counts are computed without materializing join tables. Contingency table algebra is a new extension of relational algebra, that facilitates the efficient implementation of this M\"obius virtual join operation. The M\"obius Join scales to large datasets (over 1M tuples) with complex schemas. Empirical evaluation with seven benchmark datasets showed that information about the presence and absence of links can be exploited in feature selection, association rule mining, and Bayesian network learning.