Space efficient implementation of hypergraph dualization in the D-basis algorithm

TL;DR

This paper introduces a space-efficient implementation of the D-basis algorithm that focuses on computing attribute frequencies rather than full implications, significantly reducing memory usage in data analysis.

Contribution

The new Small Space implementation reduces memory requirements by focusing on attribute frequencies, improving efficiency over previous versions of the D-basis algorithm.

Findings

Significantly lower memory usage compared to previous implementation

Comparable runtime performance in attribute frequency computation

Effective for large datasets in data analysis applications

Abstract

We present a new implementation of the $D$-basis algorithm called the Small Space which considerably reduces the algorithm's memory usage for data analysis applications. The previous implementation delivers the complete set of implications that hold on the set of attributes of an input binary table. In the new version, the only output is the frequencies of attributes that appear in the antecedents of implications from the $D$-basis, with a fixed consequent attribute. Such frequencies, rather than the implications themselves, became the primary focus in analysis of datasets where the $D$-basis has been applied over the last decade. The $D$-basis employs a hypergraph dualization algorithm, and a dualization implementation known as Reverse Search allows for the gradual computation of frequencies without the need for storing all discovered implications. We demonstrate the effectiveness of the Small Space implementation by comparing the runtimes and maximum memory usage of this new version with the current implementation.